Sep 10 14:19:40.276225 ip-10-0-138-102 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 14:19:40.669712 ip-10-0-138-102 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 14:19:40.669712 ip-10-0-138-102 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 14:19:40.669712 ip-10-0-138-102 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 14:19:40.669712 ip-10-0-138-102 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 14:19:40.669712 ip-10-0-138-102 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 14:19:40.671656 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.671571    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 14:19:40.680804 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680778    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:19:40.680804 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680799    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:19:40.680804 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680802    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:19:40.680804 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680806    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:19:40.680804 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680810    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680814    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680817    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680820    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680824    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680829    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680832    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680834    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680837    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680840    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680842    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680845    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680848    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680851    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680853    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680856    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680859    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680861    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680864    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:19:40.681033 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680867    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680870    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680877    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680881    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680883    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680886    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680889    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680891    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680894    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680897    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680899    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680902    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680904    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680907    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680910    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680913    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680915    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680918    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680921    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:19:40.681481 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680925    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680927    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680930    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680933    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680936    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680940    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680957    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680960    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680963    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680966    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680969    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680971    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680974    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680976    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680979    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680982    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680985    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680988    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680990    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:19:40.681966 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680993    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680995    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.680998    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681000    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681003    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681005    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681008    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681012    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681016    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681021    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681024    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681028    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681030    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681033    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681036    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681039    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681042    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681045    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681048    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681050    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:19:40.682467 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681053    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681057    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681061    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681063    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681066    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681069    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681071    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681074    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681077    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681079    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681082    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681085    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681087    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681090    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681093    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681095    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681098    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681101    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681816    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681823    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:19:40.682957 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681826    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681829    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681832    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681835    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681837    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681840    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681843    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681845    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681848    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681851    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681853    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681856    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681859    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681863    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681867    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681871    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681875    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681878    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681881    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:19:40.683522 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681883    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681886    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681888    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681891    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681894    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681904    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681908    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681911    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681914    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681917    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681919    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681923    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681925    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681928    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681931    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681934    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681936    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681939    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681941    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681958    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:19:40.684001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681960    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681963    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681966    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681968    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681971    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681974    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681977    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681981    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681985    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681989    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681991    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681994    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.681999    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682002    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682005    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682007    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682010    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682013    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682016    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:19:40.684480 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682019    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682021    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682024    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682026    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682029    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682031    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682034    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682036    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682039    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682042    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682044    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682047    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682050    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682052    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682054    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682057    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682059    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682062    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682064    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682067    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:19:40.684937 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682070    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682074    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682076    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682079    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682082    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682084    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682087    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682089    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682092    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682095    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682097    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682100    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682106    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682108    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682111    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682114    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682116    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682119    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682122    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682505    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:19:40.685438 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682510    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682513    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682516    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682518    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682521    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682524    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682527    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682529    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682532    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682534    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682537    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682539    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682542    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682544    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682547    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682551    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682554    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682557    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682560    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:19:40.685936 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682563    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682565    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682568    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682571    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682573    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682577    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682580    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682582    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682585    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682588    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682590    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682593    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682595    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682597    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682600    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682603    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682605    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682608    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682610    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682613    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682615    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:19:40.686414 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682617    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682620    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682622    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682625    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682627    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682630    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682632    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682635    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682638    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682641    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682643    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682646    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682649    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682652    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682654    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682657    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682661    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682664    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682666    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:19:40.686915 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682670    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682674    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682677    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682680    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682683    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682685    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682688    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682690    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682693    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682696    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682698    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682701    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682703    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682706    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682708    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682711    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682713    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682716    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682718    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682721    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:19:40.687418 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682723    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682727    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682730    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682733    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682736    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682738    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682741    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682743    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682746    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682749    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682752    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682755    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682757    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682760    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682763    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682765    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682768    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682771    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.682774    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682850    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 14:19:40.687897 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682884    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682892    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682896    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682901    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682904    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682908    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682912    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682915    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682919    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682923    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682926    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682929    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682932    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682935    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682938    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682941    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682960    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682964    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682968    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682971    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682974    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682978    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682982    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 14:19:40.688397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682985    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682988    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682991    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682994    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.682997    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683000    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683004    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683006    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683010    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683013    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683016    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683019    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683022    2580 flags.go:64] FLAG: --enable-server="true"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683024    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683032    2580 flags.go:64] FLAG: --event-burst="100"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683035    2580 flags.go:64] FLAG: --event-qps="50"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683038    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683041    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683044    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683047    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683051    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683053    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683056    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683059    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683062    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 14:19:40.688940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683068    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683071    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683074    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683077    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683080    2580 flags.go:64] FLAG: --feature-gates=""
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683084    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683087    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683090    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683093    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683096    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683099    2580 flags.go:64] FLAG: --help="false"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683102    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-138-102.ec2.internal"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683106    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683109    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683112    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683115    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683119    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683122    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683125    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683127    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683130    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683133    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683136    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683139    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 10 14:19:40.689578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683142    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683145    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683148    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683150    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683153    2580 flags.go:64] FLAG: --lock-file=""
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683156    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683159    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683161    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683168    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683171    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683174    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683177    2580 flags.go:64] FLAG: --logging-format="text"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683180    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683183    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683186    2580 flags.go:64] FLAG: --manifest-url=""
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683189    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683193    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683196    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683199    2580 flags.go:64] FLAG: --max-pods="110"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683202    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683205    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683208    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683211    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683213    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683216    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 14:19:40.690203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683219    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683227    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683230    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683233    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683236    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683238    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683245    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683248    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683251    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683253    2580 flags.go:64] FLAG: --port="10250"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683256    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683259    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b7fc83bb068593cf"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683262    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683265    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683268    2580 flags.go:64] FLAG: --register-node="true"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683271    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683276    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683279    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683282    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683285    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683288    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683291    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683294    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683297    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683300    2580 flags.go:64] FLAG: --runonce="false"
Sep 10 14:19:40.690848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683303    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683306    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683308    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683311    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683314    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683317    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683320    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683323    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683326    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683329    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683332    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683334    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683337    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683340    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683343    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683349    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683351    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683354    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683358    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683361    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683363    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683367    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683372    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683375    2580 flags.go:64] FLAG: --v="2"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683379    2580 flags.go:64] FLAG: --version="false"
Sep 10 14:19:40.691480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683384    2580 flags.go:64] FLAG: --vmodule=""
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683388    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.683391    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683509    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683513    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683516    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683519    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683521    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683524    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683527    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683530    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683533    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683535    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683538    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683541    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683544    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683546    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683549    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683551    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683553    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683556    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:19:40.692110 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683558    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683561    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683564    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683566    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683569    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683572    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683574    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683578    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683582    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683584    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683587    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683589    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683593    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683595    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683598    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683600    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683603    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683606    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683608    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683611    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:19:40.692624 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683614    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683616    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683618    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683621    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683624    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683626    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683630    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683635    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683638    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683642    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683645    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683647    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683651    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683653    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683656    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683658    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683661    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683663    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683666    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:19:40.693150 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683670    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683674    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683677    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683679    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683682    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683684    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683687    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683692    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683697    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683702    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683706    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683710    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683713    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683716    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683719    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683722    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683724    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683727    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683731    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:19:40.693616 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683733    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683736    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683739    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683741    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683744    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683747    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683750    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683752    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683755    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683757    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683760    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683762    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683765    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683769    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683773    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683776    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683779    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683781    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683784    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683786    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:19:40.694136 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683789    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:19:40.694628 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683792    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:19:40.694628 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.683795    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:19:40.694628 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.684744    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 14:19:40.694628 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.691583    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 14:19:40.694628 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.691600    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 14:19:40.694628 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.693277    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 14:19:40.694628 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.693287    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 14:19:40.694628 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693345    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:19:40.694628 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693350    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:19:40.694628 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693354    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:19:40.694628 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693358    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:19:40.694628 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693361    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:19:40.694628 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693363    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:19:40.694628 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693366    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693369    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693372    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693374    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693377    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693379    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693382    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693387    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693391    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693394    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693397    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693400    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693403    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693405    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693408    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693411    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693414    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693417    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693420    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:19:40.694996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693423    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693426    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693429    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693432    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693435    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693439    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693442    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693444    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693447    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693449    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693453    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693457    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693460    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693463    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693466    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693468    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693471    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693474    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693476    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:19:40.695468 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693479    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693481    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693484    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693487    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693489    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693492    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693494    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693497    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693499    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693502    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693505    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693508    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693510    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693513    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693516    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693519    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693522    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693525    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693528    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693531    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:19:40.695932 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693533    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693536    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693538    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693541    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693545    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693548    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693551    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693554    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693557    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693559    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693562    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693565    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693567    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693570    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693572    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693575    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693577    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693580    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693582    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693585    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:19:40.696422 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693587    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:19:40.696898 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693590    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:19:40.696898 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693593    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:19:40.696898 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693595    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:19:40.696898 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693598    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:19:40.696898 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693601    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:19:40.696898 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693603    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:19:40.696898 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693606    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:19:40.696898 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693608    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:19:40.696898 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693611    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:19:40.696898 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693613    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:19:40.696898 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693616    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:19:40.696898 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693619    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:19:40.696898 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693622    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:19:40.696898 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693625    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:19:40.696898 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.693630    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693744    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693750    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693753    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693756    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693758    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693761    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693764    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693766    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693769    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693771    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693774    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693777    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693779    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693782    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693785    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693787    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693790    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693793    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:19:40.697292 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693795    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693798    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693802    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693806    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693809    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693812    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693815    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693817    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693820    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693823    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693826    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693829    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693831    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693834    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693836    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693839    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693842    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693844    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693847    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693849    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:19:40.697745 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693852    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693854    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693857    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693860    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693863    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693867    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693869    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693872    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693874    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693877    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693879    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693882    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693885    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693887    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693890    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693893    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693895    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693908    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693911    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:19:40.698257 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693913    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693916    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693919    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693921    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693925    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693927    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693930    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693933    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693935    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693938    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693940    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693943    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693959    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693962    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693965    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693967    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693971    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693973    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693976    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693978    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:19:40.698711 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693981    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693985    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693988    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693991    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693994    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693996    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.693999    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.694002    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.694004    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.694007    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.694010    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.694012    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.694015    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.694017    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.694020    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.694022    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.694025    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.694028    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.694031    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:19:40.699266 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.694033    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:19:40.699742 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.694036    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:19:40.699742 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:40.694039    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:19:40.699742 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.694043    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 14:19:40.699742 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.694156    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 14:19:40.699742 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.696143    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 14:19:40.699742 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.697125    2580 server.go:1019] "Starting client certificate rotation"
Sep 10 14:19:40.699742 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.697220    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 14:19:40.699742 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.697253    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 14:19:40.699742 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.699605    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 14:19:40.700779 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.700759    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 14:19:40.713718 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.713692    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 10 14:19:40.718006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.717980    2580 log.go:25] "Validated CRI v1 image API"
Sep 10 14:19:40.719152 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.719131    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 14:19:40.722317 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.722295    2580 fs.go:135] Filesystem UUIDs: map[29f60fbf-8b30-4cee-8938-c1321a7b7e1d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7dced526-a188-4537-ab2f-02bd6aaaed6d:/dev/nvme0n1p3]
Sep 10 14:19:40.722401 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.722316    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 14:19:40.726701 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.726683    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 14:19:40.727530 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.727417    2580 manager.go:216] Machine: {Timestamp:2026-09-10 14:19:40.726075073 +0000 UTC m=+0.335628723 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098427 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c35aaefe16f5f69291d01dde94284 SystemUUID:ec2c35aa-efe1-6f5f-6929-1d01dde94284 BootID:c26ebad2-d12b-4a4b-aa4a-afbccef55faa Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b4:90:4e:59:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b4:90:4e:59:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:21:3a:69:21:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 14:19:40.727530 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.727525    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 14:19:40.727688 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.727633    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 14:19:40.730332 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.730300    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 14:19:40.730508 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.730333    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-102.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 14:19:40.731112 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.731100    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 14:19:40.731180 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.731118    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 14:19:40.731180 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.731136    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 14:19:40.731937 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.731925    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 14:19:40.732676 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.732665    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 14:19:40.736381 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.736367    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 14:19:40.736516 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.736505    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 14:19:40.736582 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.736543    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 14:19:40.736582 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.736563    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 14:19:40.736582 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.736582    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 14:19:40.736707 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.736597    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 10 14:19:40.736707 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.736612    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 14:19:40.737007 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.736994    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 14:19:40.737081 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.737017    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 14:19:40.739365 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.739349    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 14:19:40.741571 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.741557    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 14:19:40.741625 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.741587    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 14:19:40.743142 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.743128    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 14:19:40.743199 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.743148    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 14:19:40.743199 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.743154    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 14:19:40.743199 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.743166    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 14:19:40.743199 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.743174    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 14:19:40.743199 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.743179    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 14:19:40.743199 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.743193    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 14:19:40.743199 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.743198    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 14:19:40.743373 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.743205    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 14:19:40.743373 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.743211    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 14:19:40.743373 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.743224    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 14:19:40.743373 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.743230    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 14:19:40.744005 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.743995    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 14:19:40.744005 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.744004    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 14:19:40.745670 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.745629    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-102.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 14:19:40.745730 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:40.745682    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-102.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 14:19:40.745786 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:40.745762    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 14:19:40.748391 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.748371    2580 server.go:1268] "Started kubelet"
Sep 10 14:19:40.748505 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.748455    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 14:19:40.748563 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.748530    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 14:19:40.749003 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.748969    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 14:19:40.749314 ip-10-0-138-102 systemd[1]: Started Kubernetes Kubelet.
Sep 10 14:19:40.750513 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.750495    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 14:19:40.750920 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.750875    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 14:19:40.753668 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:40.753643    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 14:19:40.754078 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:40.753256    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-102.ec2.internal.18d3fb7a8bec86b4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-102.ec2.internal,UID:ip-10-0-138-102.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-102.ec2.internal,},FirstTimestamp:2026-09-10 14:19:40.748334772 +0000 UTC m=+0.357888421,LastTimestamp:2026-09-10 14:19:40.748334772 +0000 UTC m=+0.357888421,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-102.ec2.internal,}"
Sep 10 14:19:40.756295 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.756279    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 14:19:40.756295 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.756291    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 14:19:40.756401 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:40.756348    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:40.756852 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.756835    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 14:19:40.756852 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.756837    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 14:19:40.757020 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.756864    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 14:19:40.757020 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.756979    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 14:19:40.757020 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.756990    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 14:19:40.757183 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.757163    2580 factory.go:55] Registering systemd factory
Sep 10 14:19:40.757183 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.757184    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 10 14:19:40.757436 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.757418    2580 factory.go:153] Registering CRI-O factory
Sep 10 14:19:40.757436 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.757438    2580 factory.go:223] Registration of the crio container factory successfully
Sep 10 14:19:40.757569 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.757497    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 14:19:40.757569 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.757514    2580 factory.go:103] Registering Raw factory
Sep 10 14:19:40.757569 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.757532    2580 manager.go:1196] Started watching for new ooms in manager
Sep 10 14:19:40.758237 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.758222    2580 manager.go:349] Starting recovery of all containers
Sep 10 14:19:40.763291 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.763266    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gd8pn"
Sep 10 14:19:40.768587 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:40.768547    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 14:19:40.768745 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:40.768702    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-102.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 14:19:40.769128 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.769106    2580 manager.go:354] Recovery completed
Sep 10 14:19:40.770571 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.770544    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gd8pn"
Sep 10 14:19:40.776219 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.776050    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:19:40.778005 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.777983    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:19:40.778086 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.778018    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:19:40.778086 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.778028    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:19:40.778639 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.778614    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 14:19:40.778716 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.778640    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 14:19:40.778716 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.778658    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 14:19:40.778802 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.778625    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:19:40.779515 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.779497    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:19:40.779582 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.779534    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:19:40.779582 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.779550    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:19:40.780644 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.780629    2580 policy_none.go:49] "None policy: Start"
Sep 10 14:19:40.780644 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.780645    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 14:19:40.780735 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.780655    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 14:19:40.782584 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.782572    2580 policy_none.go:47] "Start"
Sep 10 14:19:40.834238 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.825583    2580 manager.go:337] "Starting Device Plugin manager"
Sep 10 14:19:40.834238 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:40.825630    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 14:19:40.834238 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.825640    2580 server.go:87] "Starting device plugin registration server"
Sep 10 14:19:40.834238 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.826000    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 14:19:40.834238 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.826011    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 14:19:40.834238 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.826119    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 14:19:40.834238 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.826210    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 14:19:40.834238 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.826218    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 14:19:40.834238 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:40.826788    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 14:19:40.834238 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:40.826842    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:40.883989 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.883954    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 14:19:40.885199 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.885181    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 14:19:40.885265 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.885240    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 14:19:40.885313 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.885267    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 14:19:40.885414 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:40.885377    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 14:19:40.887311 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.887291    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:19:40.926355 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.926249    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:19:40.927538 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.927523    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:19:40.927617 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.927552    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:19:40.927617 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.927570    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:19:40.927617 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.927599    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-102.ec2.internal"
Sep 10 14:19:40.934799 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.934785    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-102.ec2.internal"
Sep 10 14:19:40.934859 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:40.934807    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-102.ec2.internal\": node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:40.944355 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:40.944335    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:40.986021 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.985996    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal"]
Sep 10 14:19:40.986137 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.986058    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:19:40.987560 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.987534    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:19:40.987677 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.987567    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:19:40.987677 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.987581    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:19:40.989810 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.989794    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:19:40.989928 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.989914    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal"
Sep 10 14:19:40.989996 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.989961    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:19:40.990593 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.990574    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:19:40.990686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.990604    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:19:40.990686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.990614    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:19:40.990686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.990581    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:19:40.990686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.990682    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:19:40.990874 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.990699    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:19:40.992645 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.992630    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal"
Sep 10 14:19:40.992706 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.992653    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:19:40.993319 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.993307    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:19:40.993396 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.993346    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:19:40.993396 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:40.993360    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:19:41.011500 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:41.011479    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-102.ec2.internal\" not found" node="ip-10-0-138-102.ec2.internal"
Sep 10 14:19:41.014942 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:41.014927    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-102.ec2.internal\" not found" node="ip-10-0-138-102.ec2.internal"
Sep 10 14:19:41.044862 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:41.044838    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:41.058140 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:41.058115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e416b0faba17117313669ffa5d00f48b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-102.ec2.internal\" (UID: \"e416b0faba17117313669ffa5d00f48b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal"
Sep 10 14:19:41.058224 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:41.058144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dab447291d2c7d1953740561c77a79c0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal\" (UID: \"dab447291d2c7d1953740561c77a79c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal"
Sep 10 14:19:41.058224 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:41.058161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dab447291d2c7d1953740561c77a79c0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal\" (UID: \"dab447291d2c7d1953740561c77a79c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal"
Sep 10 14:19:41.145123 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:41.145079    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:41.158377 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:41.158348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dab447291d2c7d1953740561c77a79c0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal\" (UID: \"dab447291d2c7d1953740561c77a79c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal"
Sep 10 14:19:41.158456 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:41.158381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e416b0faba17117313669ffa5d00f48b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-102.ec2.internal\" (UID: \"e416b0faba17117313669ffa5d00f48b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal"
Sep 10 14:19:41.158456 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:41.158399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dab447291d2c7d1953740561c77a79c0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal\" (UID: \"dab447291d2c7d1953740561c77a79c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal"
Sep 10 14:19:41.158517 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:41.158455    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dab447291d2c7d1953740561c77a79c0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal\" (UID: \"dab447291d2c7d1953740561c77a79c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal"
Sep 10 14:19:41.158550 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:41.158527    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dab447291d2c7d1953740561c77a79c0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal\" (UID: \"dab447291d2c7d1953740561c77a79c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal"
Sep 10 14:19:41.158585 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:41.158568    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e416b0faba17117313669ffa5d00f48b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-102.ec2.internal\" (UID: \"e416b0faba17117313669ffa5d00f48b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal"
Sep 10 14:19:42.664047 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:42.664015    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:42.731534 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:42.731480    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal"
Sep 10 14:19:42.736422 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:42.736406    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal"
Sep 10 14:19:42.765118 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:42.765087    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:42.865609 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:42.865564    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:42.966179 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:42.966097    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:42.994008 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:42.993964    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:19:43.066804 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:43.066770    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:43.115290 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:43.115249    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 14:19:43.115726 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:43.115423    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 14:19:43.115726 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:43.115446    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 14:19:43.167293 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:43.167267    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:43.193226 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:43.193183    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 14:14:42 +0000 UTC" deadline="2028-03-31 13:15:51.249666617 +0000 UTC"
Sep 10 14:19:43.193226 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:43.193216    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13630h56m8.056453605s"
Sep 10 14:19:43.267594 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:43.267509    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:43.368053 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:43.368020    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:43.468527 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:43.468490    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:43.533746 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:43.533697    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab447291d2c7d1953740561c77a79c0.slice/crio-6686bbfd0f60cc9c25dfe33673f9b5d2fee2c5dfe2e389dc690cc182aac9263b WatchSource:0}: Error finding container 6686bbfd0f60cc9c25dfe33673f9b5d2fee2c5dfe2e389dc690cc182aac9263b: Status 404 returned error can't find the container with id 6686bbfd0f60cc9c25dfe33673f9b5d2fee2c5dfe2e389dc690cc182aac9263b
Sep 10 14:19:43.533996 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:43.533956    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode416b0faba17117313669ffa5d00f48b.slice/crio-1420d928cb18aef9dd7e374a0d466df5e6c0d43fbc98ee053db1ef52bfc02da7 WatchSource:0}: Error finding container 1420d928cb18aef9dd7e374a0d466df5e6c0d43fbc98ee053db1ef52bfc02da7: Status 404 returned error can't find the container with id 1420d928cb18aef9dd7e374a0d466df5e6c0d43fbc98ee053db1ef52bfc02da7
Sep 10 14:19:43.538276 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:43.538261    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:19:43.569592 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:43.569562    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:43.621169 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:43.621144    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:19:43.670324 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:43.670292    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found"
Sep 10 14:19:43.755619 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:43.755595    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:19:43.774788 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:43.774731    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal"
Sep 10 14:19:43.785059 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:43.785043    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 14:19:43.786079 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:43.786066    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal"
Sep 10 14:19:43.794462 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:43.794448    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 14:19:44.155797 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.155723    2580 apiserver.go:52] "Watching apiserver"
Sep 10 14:19:44.163685 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.163649    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 14:19:44.164026 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.163997    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-mtztw","openshift-ovn-kubernetes/ovnkube-node-rjjh6","kube-system/global-pull-secret-syncer-hgkhl","kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm","openshift-cluster-node-tuning-operator/tuned-wwmw8","openshift-image-registry/node-ca-4bwvd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal","openshift-multus/multus-additional-cni-plugins-nf6ht","openshift-multus/network-metrics-daemon-qvxlq","kube-system/konnectivity-agent-n82cf","openshift-dns/node-resolver-vjqdq","openshift-multus/multus-47x2k","openshift-network-diagnostics/network-check-target-hn9m5"]
Sep 10 14:19:44.166498 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.166477    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:44.166602 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.166559    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:19:44.168543 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.168522    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.170669 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.170649    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 14:19:44.170769 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.170682    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 14:19:44.170881 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.170849    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:44.171001 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.170893    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 14:19:44.171001 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.170907    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:19:44.171001 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.170948    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kf4dz\""
Sep 10 14:19:44.171001 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.170955    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 14:19:44.172795 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.172776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:19:44.172881 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.172835    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:19:44.174939 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.174917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.175292 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.175262    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 14:19:44.177505 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.177100    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 14:19:44.177505 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.177176    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 14:19:44.177505 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.177249    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.177505 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.177417    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 14:19:44.177725 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.177700    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q4hsx\""
Sep 10 14:19:44.179716 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.179502    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.179716 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.179579    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 14:19:44.179716 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.179595    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m5tqk\""
Sep 10 14:19:44.179716 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.179639    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:19:44.181848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.181816    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4bwvd"
Sep 10 14:19:44.181946 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.181886    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f6gwc\""
Sep 10 14:19:44.182186 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.182166    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 14:19:44.182281 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.182270    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 14:19:44.184359 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.184341    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.185626 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8999t\" (UniqueName: \"kubernetes.io/projected/56412239-e32d-418f-b321-34de77a92ce6-kube-api-access-8999t\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.185724 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-lib-modules\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.185724 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-multus-cni-dir\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.185724 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-run-k8s-cni-cncf-io\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.185724 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185717    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.185930 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185723    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 14:19:44.185930 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-device-dir\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.185930 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-sysctl-conf\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.185930 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-systemd\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.185930 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185804    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 14:19:44.185930 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20919d23-b9fc-4552-8082-4b83053f870b-cni-binary-copy\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.185930 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-etc-kubernetes\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.185930 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.185930 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-sysconfig\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-sysctl-d\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.185960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b046f46-d48b-4025-b9f9-37c5eea82ff9-dbus\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-modprobe-d\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-run\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186054    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7dq9n\""
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186081    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-var-lib-cni-bin\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-var-lib-kubelet\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186148    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndkbz\" (UniqueName: \"kubernetes.io/projected/20919d23-b9fc-4552-8082-4b83053f870b-kube-api-access-ndkbz\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-var-lib-kubelet\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-tmp\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs\") pod \"network-metrics-daemon-qvxlq\" (UID: \"50de453c-4d08-4f49-9e7e-0d5e2221e72e\") " pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-system-cni-dir\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186297    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-multus-socket-dir-parent\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186322    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20919d23-b9fc-4552-8082-4b83053f870b-multus-daemon-config\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.186362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-socket-dir\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-sys-fs\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-sys\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-cnibin\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-multus-conf-dir\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186462    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-kubernetes\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-tuned\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186513    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc94j\" (UniqueName: \"kubernetes.io/projected/50de453c-4d08-4f49-9e7e-0d5e2221e72e-kube-api-access-rc94j\") pod \"network-metrics-daemon-qvxlq\" (UID: \"50de453c-4d08-4f49-9e7e-0d5e2221e72e\") " pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-run-netns\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186559    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b046f46-d48b-4025-b9f9-37c5eea82ff9-kubelet-config\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186591    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjqs5\" (UniqueName: \"kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5\") pod \"network-check-target-hn9m5\" (UID: \"dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33\") " pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-registration-dir\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-host\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkglc\" (UniqueName: \"kubernetes.io/projected/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-kube-api-access-rkglc\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-var-lib-cni-multus\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.187188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mtztw"
Sep 10 14:19:44.188101 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-hostroot\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.188101 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.186804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-run-multus-certs\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.188101 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.187485    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 14:19:44.188101 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.187611    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 14:19:44.188101 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.187634    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 14:19:44.188101 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.187821    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f88xw\""
Sep 10 14:19:44.190615 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.189288    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 14:19:44.190615 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.189544    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:19:44.190615 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.189828    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 14:19:44.190615 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.190207    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 14:19:44.190937 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.190748    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n82cf"
Sep 10 14:19:44.190937 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.190864    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-65vj5\""
Sep 10 14:19:44.192906 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.192679    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vjqdq"
Sep 10 14:19:44.193330 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.193317    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 14:19:44.193432 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.193322    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 14:19:44.194172 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.194157    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7np9z\""
Sep 10 14:19:44.194914 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.194895    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 14:19:44.195208 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.195167    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hmrq4\""
Sep 10 14:19:44.195208 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.195180    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 14:19:44.207698 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.207663    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jnr74"
Sep 10 14:19:44.214656 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.214634    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jnr74"
Sep 10 14:19:44.276044 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.276019    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 14:19:44.287213 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287187    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-socket-dir\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.287363 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-cnibin\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.287363 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc94j\" (UniqueName: \"kubernetes.io/projected/50de453c-4d08-4f49-9e7e-0d5e2221e72e-kube-api-access-rc94j\") pod \"network-metrics-daemon-qvxlq\" (UID: \"50de453c-4d08-4f49-9e7e-0d5e2221e72e\") " pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:44.287363 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287286    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-run-netns\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.287363 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:44.287363 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51965344-8fbc-41c8-b3ef-2f553017edbe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.287606 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-run-openvswitch\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.287606 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-node-log\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.287606 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287388    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-socket-dir\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.287606 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-run-ovn-kubernetes\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.287606 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-registration-dir\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.287606 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkglc\" (UniqueName: \"kubernetes.io/projected/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-kube-api-access-rkglc\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.287606 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-run-multus-certs\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.287606 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-var-lib-openvswitch\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.287606 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-log-socket\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.287606 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-cni-netd\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.287606 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-cnibin\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.287606 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.287509    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7ef5dcd-b063-44f5-85ae-d9dc78521768-ovnkube-config\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-run-netns\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.287699    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret podName:0b046f46-d48b-4025-b9f9-37c5eea82ff9 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:44.787676735 +0000 UTC m=+2.979204912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret") pod "global-pull-secret-syncer-hgkhl" (UID: "0b046f46-d48b-4025-b9f9-37c5eea82ff9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77jlp\" (UniqueName: \"kubernetes.io/projected/5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3-kube-api-access-77jlp\") pod \"iptables-alerter-mtztw\" (UID: \"5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3\") " pod="openshift-network-operator/iptables-alerter-mtztw"
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-registration-dir\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8999t\" (UniqueName: \"kubernetes.io/projected/56412239-e32d-418f-b321-34de77a92ce6-kube-api-access-8999t\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-run-k8s-cni-cncf-io\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51965344-8fbc-41c8-b3ef-2f553017edbe-cnibin\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-run-multus-certs\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-run-k8s-cni-cncf-io\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-sysctl-conf\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-systemd\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.287996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-etc-kubernetes\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smh6n\" (UniqueName: \"kubernetes.io/projected/3523580c-458a-486f-9cc6-8705298c0617-kube-api-access-smh6n\") pod \"node-ca-4bwvd\" (UID: \"3523580c-458a-486f-9cc6-8705298c0617\") " pod="openshift-image-registry/node-ca-4bwvd"
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-etc-kubernetes\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.288182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288072    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-run-netns\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288123    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-systemd\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48aab852-7724-4969-b2c7-d6d2db10ec8d-hosts-file\") pod \"node-resolver-vjqdq\" (UID: \"48aab852-7724-4969-b2c7-d6d2db10ec8d\") " pod="openshift-dns/node-resolver-vjqdq"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-sysctl-conf\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-sysctl-d\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-slash\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-run\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288263    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-var-lib-cni-bin\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-systemd-units\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-var-lib-kubelet\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-tmp\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-system-cni-dir\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288388    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-var-lib-kubelet\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20919d23-b9fc-4552-8082-4b83053f870b-multus-daemon-config\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-run\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-sys-fs\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.288895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288434    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-sysctl-d\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-var-lib-cni-bin\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-system-cni-dir\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-sys\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-multus-conf-dir\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clwn9\" (UniqueName: \"kubernetes.io/projected/a7ef5dcd-b063-44f5-85ae-d9dc78521768-kube-api-access-clwn9\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-kubernetes\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-sys\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-multus-conf-dir\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-kubernetes\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-tuned\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b046f46-d48b-4025-b9f9-37c5eea82ff9-kubelet-config\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288874    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51965344-8fbc-41c8-b3ef-2f553017edbe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7ef5dcd-b063-44f5-85ae-d9dc78521768-ovn-node-metrics-cert\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/496deff1-b6df-408f-8282-2b61f4563584-agent-certs\") pod \"konnectivity-agent-n82cf\" (UID: \"496deff1-b6df-408f-8282-2b61f4563584\") " pod="kube-system/konnectivity-agent-n82cf"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqs5\" (UniqueName: \"kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5\") pod \"network-check-target-hn9m5\" (UID: \"dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33\") " pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-host\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.289425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-var-lib-cni-multus\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.288964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-hostroot\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20919d23-b9fc-4552-8082-4b83053f870b-multus-daemon-config\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51965344-8fbc-41c8-b3ef-2f553017edbe-system-cni-dir\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpz4f\" (UniqueName: \"kubernetes.io/projected/51965344-8fbc-41c8-b3ef-2f553017edbe-kube-api-access-wpz4f\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-run-ovn\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-lib-modules\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-multus-cni-dir\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51965344-8fbc-41c8-b3ef-2f553017edbe-cni-binary-copy\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51965344-8fbc-41c8-b3ef-2f553017edbe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3523580c-458a-486f-9cc6-8705298c0617-host\") pod \"node-ca-4bwvd\" (UID: \"3523580c-458a-486f-9cc6-8705298c0617\") " pod="openshift-image-registry/node-ca-4bwvd"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-cni-bin\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289189    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7ef5dcd-b063-44f5-85ae-d9dc78521768-env-overrides\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7ef5dcd-b063-44f5-85ae-d9dc78521768-ovnkube-script-lib\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289233    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b046f46-d48b-4025-b9f9-37c5eea82ff9-kubelet-config\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-host\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.290124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289291    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-multus-cni-dir\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-device-dir\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-hostroot\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-var-lib-cni-multus\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-device-dir\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56412239-e32d-418f-b321-34de77a92ce6-sys-fs\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20919d23-b9fc-4552-8082-4b83053f870b-cni-binary-copy\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3-iptables-alerter-script\") pod \"iptables-alerter-mtztw\" (UID: \"5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3\") " pod="openshift-network-operator/iptables-alerter-mtztw"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48aab852-7724-4969-b2c7-d6d2db10ec8d-tmp-dir\") pod \"node-resolver-vjqdq\" (UID: \"48aab852-7724-4969-b2c7-d6d2db10ec8d\") " pod="openshift-dns/node-resolver-vjqdq"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289478    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-lib-modules\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxds5\" (UniqueName: \"kubernetes.io/projected/48aab852-7724-4969-b2c7-d6d2db10ec8d-kube-api-access-jxds5\") pod \"node-resolver-vjqdq\" (UID: \"48aab852-7724-4969-b2c7-d6d2db10ec8d\") " pod="openshift-dns/node-resolver-vjqdq"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-sysconfig\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b046f46-d48b-4025-b9f9-37c5eea82ff9-dbus\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-run-systemd\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-modprobe-d\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.290893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-var-lib-kubelet\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndkbz\" (UniqueName: \"kubernetes.io/projected/20919d23-b9fc-4552-8082-4b83053f870b-kube-api-access-ndkbz\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-kubelet\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/496deff1-b6df-408f-8282-2b61f4563584-konnectivity-ca\") pod \"konnectivity-agent-n82cf\" (UID: \"496deff1-b6df-408f-8282-2b61f4563584\") " pod="kube-system/konnectivity-agent-n82cf"
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs\") pod \"network-metrics-daemon-qvxlq\" (UID: \"50de453c-4d08-4f49-9e7e-0d5e2221e72e\") " pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-multus-socket-dir-parent\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289724    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3523580c-458a-486f-9cc6-8705298c0617-serviceca\") pod \"node-ca-4bwvd\" (UID: \"3523580c-458a-486f-9cc6-8705298c0617\") " pod="openshift-image-registry/node-ca-4bwvd"
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-etc-openvswitch\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3-host-slash\") pod \"iptables-alerter-mtztw\" (UID: \"5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3\") " pod="openshift-network-operator/iptables-alerter-mtztw"
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20919d23-b9fc-4552-8082-4b83053f870b-cni-binary-copy\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.289932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-host-var-lib-kubelet\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.290031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-sysconfig\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.290092    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20919d23-b9fc-4552-8082-4b83053f870b-multus-socket-dir-parent\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.290104    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.290151    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs podName:50de453c-4d08-4f49-9e7e-0d5e2221e72e nodeName:}" failed. No retries permitted until 2026-09-10 14:19:44.790134474 +0000 UTC m=+2.981662653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs") pod "network-metrics-daemon-qvxlq" (UID: "50de453c-4d08-4f49-9e7e-0d5e2221e72e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.290169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b046f46-d48b-4025-b9f9-37c5eea82ff9-dbus\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:44.291686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.290226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-modprobe-d\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.293671 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.293495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-etc-tuned\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.293671 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.293639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-tmp\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.295388 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.295370    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:19:44.295500 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.295391    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:19:44.295500 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.295404    2580 projected.go:196] Error preparing data for projected volume kube-api-access-fjqs5 for pod openshift-network-diagnostics/network-check-target-hn9m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:44.295500 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.295454    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5 podName:dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:44.795438494 +0000 UTC m=+2.986966683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fjqs5" (UniqueName: "kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5") pod "network-check-target-hn9m5" (UID: "dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:44.297942 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.297919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8999t\" (UniqueName: \"kubernetes.io/projected/56412239-e32d-418f-b321-34de77a92ce6-kube-api-access-8999t\") pod \"aws-ebs-csi-driver-node-q4zwm\" (UID: \"56412239-e32d-418f-b321-34de77a92ce6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.298675 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.298653    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc94j\" (UniqueName: \"kubernetes.io/projected/50de453c-4d08-4f49-9e7e-0d5e2221e72e-kube-api-access-rc94j\") pod \"network-metrics-daemon-qvxlq\" (UID: \"50de453c-4d08-4f49-9e7e-0d5e2221e72e\") " pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:44.298876 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.298852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkglc\" (UniqueName: \"kubernetes.io/projected/2c791935-61bc-4e51-a9b6-fdc8076ee4c6-kube-api-access-rkglc\") pod \"tuned-wwmw8\" (UID: \"2c791935-61bc-4e51-a9b6-fdc8076ee4c6\") " pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.299750 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.299728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndkbz\" (UniqueName: \"kubernetes.io/projected/20919d23-b9fc-4552-8082-4b83053f870b-kube-api-access-ndkbz\") pod \"multus-47x2k\" (UID: \"20919d23-b9fc-4552-8082-4b83053f870b\") " pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.308911 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.308867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" event={"ID":"dab447291d2c7d1953740561c77a79c0","Type":"ContainerStarted","Data":"6686bbfd0f60cc9c25dfe33673f9b5d2fee2c5dfe2e389dc690cc182aac9263b"}
Sep 10 14:19:44.311241 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.311216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal" event={"ID":"e416b0faba17117313669ffa5d00f48b","Type":"ContainerStarted","Data":"1420d928cb18aef9dd7e374a0d466df5e6c0d43fbc98ee053db1ef52bfc02da7"}
Sep 10 14:19:44.390487 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390452    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3523580c-458a-486f-9cc6-8705298c0617-serviceca\") pod \"node-ca-4bwvd\" (UID: \"3523580c-458a-486f-9cc6-8705298c0617\") " pod="openshift-image-registry/node-ca-4bwvd"
Sep 10 14:19:44.390665 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-etc-openvswitch\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.390665 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.390665 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3-host-slash\") pod \"iptables-alerter-mtztw\" (UID: \"5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3\") " pod="openshift-network-operator/iptables-alerter-mtztw"
Sep 10 14:19:44.390665 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51965344-8fbc-41c8-b3ef-2f553017edbe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.390665 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-run-openvswitch\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.390665 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-node-log\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.390665 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390628    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-etc-openvswitch\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.390665 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-run-ovn-kubernetes\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390684    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-run-ovn-kubernetes\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-var-lib-openvswitch\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-log-socket\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-var-lib-openvswitch\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-cni-netd\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7ef5dcd-b063-44f5-85ae-d9dc78521768-ovnkube-config\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jlp\" (UniqueName: \"kubernetes.io/projected/5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3-kube-api-access-77jlp\") pod \"iptables-alerter-mtztw\" (UID: \"5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3\") " pod="openshift-network-operator/iptables-alerter-mtztw"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51965344-8fbc-41c8-b3ef-2f553017edbe-cnibin\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smh6n\" (UniqueName: \"kubernetes.io/projected/3523580c-458a-486f-9cc6-8705298c0617-kube-api-access-smh6n\") pod \"node-ca-4bwvd\" (UID: \"3523580c-458a-486f-9cc6-8705298c0617\") " pod="openshift-image-registry/node-ca-4bwvd"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-run-netns\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51965344-8fbc-41c8-b3ef-2f553017edbe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48aab852-7724-4969-b2c7-d6d2db10ec8d-hosts-file\") pod \"node-resolver-vjqdq\" (UID: \"48aab852-7724-4969-b2c7-d6d2db10ec8d\") " pod="openshift-dns/node-resolver-vjqdq"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-slash\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390925    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-systemd-units\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390964    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3-host-slash\") pod \"iptables-alerter-mtztw\" (UID: \"5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3\") " pod="openshift-network-operator/iptables-alerter-mtztw"
Sep 10 14:19:44.391057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.390999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clwn9\" (UniqueName: \"kubernetes.io/projected/a7ef5dcd-b063-44f5-85ae-d9dc78521768-kube-api-access-clwn9\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51965344-8fbc-41c8-b3ef-2f553017edbe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-run-openvswitch\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7ef5dcd-b063-44f5-85ae-d9dc78521768-ovn-node-metrics-cert\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3523580c-458a-486f-9cc6-8705298c0617-serviceca\") pod \"node-ca-4bwvd\" (UID: \"3523580c-458a-486f-9cc6-8705298c0617\") " pod="openshift-image-registry/node-ca-4bwvd"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-run-netns\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/496deff1-b6df-408f-8282-2b61f4563584-agent-certs\") pod \"konnectivity-agent-n82cf\" (UID: \"496deff1-b6df-408f-8282-2b61f4563584\") " pod="kube-system/konnectivity-agent-n82cf"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-node-log\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-cni-netd\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51965344-8fbc-41c8-b3ef-2f553017edbe-system-cni-dir\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-log-socket\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpz4f\" (UniqueName: \"kubernetes.io/projected/51965344-8fbc-41c8-b3ef-2f553017edbe-kube-api-access-wpz4f\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51965344-8fbc-41c8-b3ef-2f553017edbe-cnibin\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-run-ovn\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51965344-8fbc-41c8-b3ef-2f553017edbe-cni-binary-copy\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51965344-8fbc-41c8-b3ef-2f553017edbe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3523580c-458a-486f-9cc6-8705298c0617-host\") pod \"node-ca-4bwvd\" (UID: \"3523580c-458a-486f-9cc6-8705298c0617\") " pod="openshift-image-registry/node-ca-4bwvd"
Sep 10 14:19:44.391826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-cni-bin\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7ef5dcd-b063-44f5-85ae-d9dc78521768-env-overrides\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7ef5dcd-b063-44f5-85ae-d9dc78521768-ovnkube-script-lib\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3-iptables-alerter-script\") pod \"iptables-alerter-mtztw\" (UID: \"5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3\") " pod="openshift-network-operator/iptables-alerter-mtztw"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48aab852-7724-4969-b2c7-d6d2db10ec8d-tmp-dir\") pod \"node-resolver-vjqdq\" (UID: \"48aab852-7724-4969-b2c7-d6d2db10ec8d\") " pod="openshift-dns/node-resolver-vjqdq"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxds5\" (UniqueName: \"kubernetes.io/projected/48aab852-7724-4969-b2c7-d6d2db10ec8d-kube-api-access-jxds5\") pod \"node-resolver-vjqdq\" (UID: \"48aab852-7724-4969-b2c7-d6d2db10ec8d\") " pod="openshift-dns/node-resolver-vjqdq"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-run-systemd\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-kubelet\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/496deff1-b6df-408f-8282-2b61f4563584-konnectivity-ca\") pod \"konnectivity-agent-n82cf\" (UID: \"496deff1-b6df-408f-8282-2b61f4563584\") " pod="kube-system/konnectivity-agent-n82cf"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391575    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7ef5dcd-b063-44f5-85ae-d9dc78521768-ovnkube-config\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391634    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-slash\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48aab852-7724-4969-b2c7-d6d2db10ec8d-hosts-file\") pod \"node-resolver-vjqdq\" (UID: \"48aab852-7724-4969-b2c7-d6d2db10ec8d\") " pod="openshift-dns/node-resolver-vjqdq"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-systemd-units\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.391942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51965344-8fbc-41c8-b3ef-2f553017edbe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.392095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7ef5dcd-b063-44f5-85ae-d9dc78521768-env-overrides\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.392103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/496deff1-b6df-408f-8282-2b61f4563584-konnectivity-ca\") pod \"konnectivity-agent-n82cf\" (UID: \"496deff1-b6df-408f-8282-2b61f4563584\") " pod="kube-system/konnectivity-agent-n82cf"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.392211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51965344-8fbc-41c8-b3ef-2f553017edbe-system-cni-dir\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.392633 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.392555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51965344-8fbc-41c8-b3ef-2f553017edbe-cni-binary-copy\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.393411 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.392595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7ef5dcd-b063-44f5-85ae-d9dc78521768-ovnkube-script-lib\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.393411 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.392654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-run-ovn\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.393411 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.392965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51965344-8fbc-41c8-b3ef-2f553017edbe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.393411 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.393035    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-cni-bin\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.393411 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.393035    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3523580c-458a-486f-9cc6-8705298c0617-host\") pod \"node-ca-4bwvd\" (UID: \"3523580c-458a-486f-9cc6-8705298c0617\") " pod="openshift-image-registry/node-ca-4bwvd"
Sep 10 14:19:44.393411 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.393099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-run-systemd\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.393411 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.393250    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7ef5dcd-b063-44f5-85ae-d9dc78521768-host-kubelet\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.393411 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.393361    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48aab852-7724-4969-b2c7-d6d2db10ec8d-tmp-dir\") pod \"node-resolver-vjqdq\" (UID: \"48aab852-7724-4969-b2c7-d6d2db10ec8d\") " pod="openshift-dns/node-resolver-vjqdq"
Sep 10 14:19:44.393833 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.393765    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3-iptables-alerter-script\") pod \"iptables-alerter-mtztw\" (UID: \"5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3\") " pod="openshift-network-operator/iptables-alerter-mtztw"
Sep 10 14:19:44.394524 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.394500    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7ef5dcd-b063-44f5-85ae-d9dc78521768-ovn-node-metrics-cert\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.396490 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.396448    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/496deff1-b6df-408f-8282-2b61f4563584-agent-certs\") pod \"konnectivity-agent-n82cf\" (UID: \"496deff1-b6df-408f-8282-2b61f4563584\") " pod="kube-system/konnectivity-agent-n82cf"
Sep 10 14:19:44.401778 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.401749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpz4f\" (UniqueName: \"kubernetes.io/projected/51965344-8fbc-41c8-b3ef-2f553017edbe-kube-api-access-wpz4f\") pod \"multus-additional-cni-plugins-nf6ht\" (UID: \"51965344-8fbc-41c8-b3ef-2f553017edbe\") " pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.403016 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.402959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxds5\" (UniqueName: \"kubernetes.io/projected/48aab852-7724-4969-b2c7-d6d2db10ec8d-kube-api-access-jxds5\") pod \"node-resolver-vjqdq\" (UID: \"48aab852-7724-4969-b2c7-d6d2db10ec8d\") " pod="openshift-dns/node-resolver-vjqdq"
Sep 10 14:19:44.404235 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.403996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smh6n\" (UniqueName: \"kubernetes.io/projected/3523580c-458a-486f-9cc6-8705298c0617-kube-api-access-smh6n\") pod \"node-ca-4bwvd\" (UID: \"3523580c-458a-486f-9cc6-8705298c0617\") " pod="openshift-image-registry/node-ca-4bwvd"
Sep 10 14:19:44.404235 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.404195    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clwn9\" (UniqueName: \"kubernetes.io/projected/a7ef5dcd-b063-44f5-85ae-d9dc78521768-kube-api-access-clwn9\") pod \"ovnkube-node-rjjh6\" (UID: \"a7ef5dcd-b063-44f5-85ae-d9dc78521768\") " pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.404840 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.404817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77jlp\" (UniqueName: \"kubernetes.io/projected/5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3-kube-api-access-77jlp\") pod \"iptables-alerter-mtztw\" (UID: \"5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3\") " pod="openshift-network-operator/iptables-alerter-mtztw"
Sep 10 14:19:44.479950 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.479857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-47x2k"
Sep 10 14:19:44.490336 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.490314    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm"
Sep 10 14:19:44.494570 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:44.494540    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20919d23_b9fc_4552_8082_4b83053f870b.slice/crio-9057c037049b1b50f0f66419397bb7efe3b51fd38f27b0443d876ed4e8f5252c WatchSource:0}: Error finding container 9057c037049b1b50f0f66419397bb7efe3b51fd38f27b0443d876ed4e8f5252c: Status 404 returned error can't find the container with id 9057c037049b1b50f0f66419397bb7efe3b51fd38f27b0443d876ed4e8f5252c
Sep 10 14:19:44.500511 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.500491    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wwmw8"
Sep 10 14:19:44.504747 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:44.504708    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56412239_e32d_418f_b321_34de77a92ce6.slice/crio-c230546b1e36c7b4daea0af77dac8cb272cff89914dbada03a5d09fe05a8906b WatchSource:0}: Error finding container c230546b1e36c7b4daea0af77dac8cb272cff89914dbada03a5d09fe05a8906b: Status 404 returned error can't find the container with id c230546b1e36c7b4daea0af77dac8cb272cff89914dbada03a5d09fe05a8906b
Sep 10 14:19:44.507640 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.507405    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n82cf"
Sep 10 14:19:44.510681 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.510656    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:19:44.512933 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:44.512914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c791935_61bc_4e51_a9b6_fdc8076ee4c6.slice/crio-839cdc150d8c5dde7200ad9e4cd879f2bdb26fa6daef38aef4ae615476214bc3 WatchSource:0}: Error finding container 839cdc150d8c5dde7200ad9e4cd879f2bdb26fa6daef38aef4ae615476214bc3: Status 404 returned error can't find the container with id 839cdc150d8c5dde7200ad9e4cd879f2bdb26fa6daef38aef4ae615476214bc3
Sep 10 14:19:44.514077 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.513914    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nf6ht"
Sep 10 14:19:44.518128 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:44.518102    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496deff1_b6df_408f_8282_2b61f4563584.slice/crio-7583430fe1e1928917fb6fde3737b52c4f842d79965523d7f95af7d497092f4a WatchSource:0}: Error finding container 7583430fe1e1928917fb6fde3737b52c4f842d79965523d7f95af7d497092f4a: Status 404 returned error can't find the container with id 7583430fe1e1928917fb6fde3737b52c4f842d79965523d7f95af7d497092f4a
Sep 10 14:19:44.522602 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.522424    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4bwvd"
Sep 10 14:19:44.528610 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.528403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:19:44.529157 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:44.529129    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51965344_8fbc_41c8_b3ef_2f553017edbe.slice/crio-74a5b2fabe793e193b992f93d9be5573dc1a6cebca05e5366bbf81d6fbe8dcf6 WatchSource:0}: Error finding container 74a5b2fabe793e193b992f93d9be5573dc1a6cebca05e5366bbf81d6fbe8dcf6: Status 404 returned error can't find the container with id 74a5b2fabe793e193b992f93d9be5573dc1a6cebca05e5366bbf81d6fbe8dcf6
Sep 10 14:19:44.534796 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:44.534767    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3523580c_458a_486f_9cc6_8705298c0617.slice/crio-ecd3aeb80f33c77bfb726bb0bc32587cc2941bf634e55e726c34b3ff0fc179ab WatchSource:0}: Error finding container ecd3aeb80f33c77bfb726bb0bc32587cc2941bf634e55e726c34b3ff0fc179ab: Status 404 returned error can't find the container with id ecd3aeb80f33c77bfb726bb0bc32587cc2941bf634e55e726c34b3ff0fc179ab
Sep 10 14:19:44.537215 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.537200    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mtztw"
Sep 10 14:19:44.539546 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:44.539523    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ef5dcd_b063_44f5_85ae_d9dc78521768.slice/crio-fbbb9a8547592429dc386a193d6cc81a64182e36ac79d3dd37df7b01530c2179 WatchSource:0}: Error finding container fbbb9a8547592429dc386a193d6cc81a64182e36ac79d3dd37df7b01530c2179: Status 404 returned error can't find the container with id fbbb9a8547592429dc386a193d6cc81a64182e36ac79d3dd37df7b01530c2179
Sep 10 14:19:44.542395 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.542222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vjqdq"
Sep 10 14:19:44.553332 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:44.553306    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4523e7_b3bf_40bb_90f5_40b2a7b26ce3.slice/crio-b2409a37310e06ff3de8f65657a31e390dfd20a7cbff34ff2fe5f3f6f4aaf48c WatchSource:0}: Error finding container b2409a37310e06ff3de8f65657a31e390dfd20a7cbff34ff2fe5f3f6f4aaf48c: Status 404 returned error can't find the container with id b2409a37310e06ff3de8f65657a31e390dfd20a7cbff34ff2fe5f3f6f4aaf48c
Sep 10 14:19:44.554307 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:19:44.554278    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48aab852_7724_4969_b2c7_d6d2db10ec8d.slice/crio-b7df6b01f9e83657dc569cf46939c724eeb7f3cd93e8625b179c9b0733bad97a WatchSource:0}: Error finding container b7df6b01f9e83657dc569cf46939c724eeb7f3cd93e8625b179c9b0733bad97a: Status 404 returned error can't find the container with id b7df6b01f9e83657dc569cf46939c724eeb7f3cd93e8625b179c9b0733bad97a
Sep 10 14:19:44.794736 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.794653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:44.794893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.794748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs\") pod \"network-metrics-daemon-qvxlq\" (UID: \"50de453c-4d08-4f49-9e7e-0d5e2221e72e\") " pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:44.794893 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.794833    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:44.794893 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.794847    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:44.795059 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.794898    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret podName:0b046f46-d48b-4025-b9f9-37c5eea82ff9 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:45.794879912 +0000 UTC m=+3.986408093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret") pod "global-pull-secret-syncer-hgkhl" (UID: "0b046f46-d48b-4025-b9f9-37c5eea82ff9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:44.795059 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.794918    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs podName:50de453c-4d08-4f49-9e7e-0d5e2221e72e nodeName:}" failed. No retries permitted until 2026-09-10 14:19:45.794908302 +0000 UTC m=+3.986436479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs") pod "network-metrics-daemon-qvxlq" (UID: "50de453c-4d08-4f49-9e7e-0d5e2221e72e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:44.895408 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:44.895374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqs5\" (UniqueName: \"kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5\") pod \"network-check-target-hn9m5\" (UID: \"dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33\") " pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:19:44.895579 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.895541    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:19:44.895579 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.895564    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:19:44.895579 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.895573    2580 projected.go:196] Error preparing data for projected volume kube-api-access-fjqs5 for pod openshift-network-diagnostics/network-check-target-hn9m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:44.895690 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:44.895615    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5 podName:dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:45.895600114 +0000 UTC m=+4.087128290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjqs5" (UniqueName: "kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5") pod "network-check-target-hn9m5" (UID: "dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:45.218395 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.218267    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:14:44 +0000 UTC" deadline="2028-04-06 13:07:42.155168629 +0000 UTC"
Sep 10 14:19:45.218395 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.218315    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13774h47m56.936858164s"
Sep 10 14:19:45.305121 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.304475    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:45.305121 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:45.304601    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:19:45.305121 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.304687    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:45.305121 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:45.304779    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:19:45.315856 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.315784    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf6ht" event={"ID":"51965344-8fbc-41c8-b3ef-2f553017edbe","Type":"ContainerStarted","Data":"74a5b2fabe793e193b992f93d9be5573dc1a6cebca05e5366bbf81d6fbe8dcf6"}
Sep 10 14:19:45.317391 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.317361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-47x2k" event={"ID":"20919d23-b9fc-4552-8082-4b83053f870b","Type":"ContainerStarted","Data":"9057c037049b1b50f0f66419397bb7efe3b51fd38f27b0443d876ed4e8f5252c"}
Sep 10 14:19:45.320568 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.320535    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vjqdq" event={"ID":"48aab852-7724-4969-b2c7-d6d2db10ec8d","Type":"ContainerStarted","Data":"b7df6b01f9e83657dc569cf46939c724eeb7f3cd93e8625b179c9b0733bad97a"}
Sep 10 14:19:45.322129 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.322094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mtztw" event={"ID":"5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3","Type":"ContainerStarted","Data":"b2409a37310e06ff3de8f65657a31e390dfd20a7cbff34ff2fe5f3f6f4aaf48c"}
Sep 10 14:19:45.323744 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.323721    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4bwvd" event={"ID":"3523580c-458a-486f-9cc6-8705298c0617","Type":"ContainerStarted","Data":"ecd3aeb80f33c77bfb726bb0bc32587cc2941bf634e55e726c34b3ff0fc179ab"}
Sep 10 14:19:45.325193 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.325167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n82cf" event={"ID":"496deff1-b6df-408f-8282-2b61f4563584","Type":"ContainerStarted","Data":"7583430fe1e1928917fb6fde3737b52c4f842d79965523d7f95af7d497092f4a"}
Sep 10 14:19:45.326640 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.326616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wwmw8" event={"ID":"2c791935-61bc-4e51-a9b6-fdc8076ee4c6","Type":"ContainerStarted","Data":"839cdc150d8c5dde7200ad9e4cd879f2bdb26fa6daef38aef4ae615476214bc3"}
Sep 10 14:19:45.328098 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.328076    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm" event={"ID":"56412239-e32d-418f-b321-34de77a92ce6","Type":"ContainerStarted","Data":"c230546b1e36c7b4daea0af77dac8cb272cff89914dbada03a5d09fe05a8906b"}
Sep 10 14:19:45.330562 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.330480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6" event={"ID":"a7ef5dcd-b063-44f5-85ae-d9dc78521768","Type":"ContainerStarted","Data":"fbbb9a8547592429dc386a193d6cc81a64182e36ac79d3dd37df7b01530c2179"}
Sep 10 14:19:45.802251 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.802193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs\") pod \"network-metrics-daemon-qvxlq\" (UID: \"50de453c-4d08-4f49-9e7e-0d5e2221e72e\") " pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:45.802417 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.802257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:45.802417 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:45.802394    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:45.803289 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:45.802782    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret podName:0b046f46-d48b-4025-b9f9-37c5eea82ff9 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:47.802759631 +0000 UTC m=+5.994287811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret") pod "global-pull-secret-syncer-hgkhl" (UID: "0b046f46-d48b-4025-b9f9-37c5eea82ff9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:45.803289 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:45.803183    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:45.803289 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:45.803243    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs podName:50de453c-4d08-4f49-9e7e-0d5e2221e72e nodeName:}" failed. No retries permitted until 2026-09-10 14:19:47.803216478 +0000 UTC m=+5.994744662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs") pod "network-metrics-daemon-qvxlq" (UID: "50de453c-4d08-4f49-9e7e-0d5e2221e72e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:45.904235 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:45.903535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqs5\" (UniqueName: \"kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5\") pod \"network-check-target-hn9m5\" (UID: \"dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33\") " pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:19:45.904235 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:45.903733    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:19:45.904235 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:45.903754    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:19:45.904235 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:45.903768    2580 projected.go:196] Error preparing data for projected volume kube-api-access-fjqs5 for pod openshift-network-diagnostics/network-check-target-hn9m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:45.904235 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:45.903827    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5 podName:dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:47.903808365 +0000 UTC m=+6.095336545 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjqs5" (UniqueName: "kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5") pod "network-check-target-hn9m5" (UID: "dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:46.032710 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:46.032664    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:19:46.218850 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:46.218765    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:14:44 +0000 UTC" deadline="2028-02-09 15:21:13.61612865 +0000 UTC"
Sep 10 14:19:46.218850 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:46.218807    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12409h1m27.397325594s"
Sep 10 14:19:46.305840 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:46.305641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:19:46.305840 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:46.305779    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:19:47.304572 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:47.304491    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:47.305025 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:47.304626    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:19:47.305088 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:47.305026    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:47.305149 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:47.305129    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:19:47.833150 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:47.833112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs\") pod \"network-metrics-daemon-qvxlq\" (UID: \"50de453c-4d08-4f49-9e7e-0d5e2221e72e\") " pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:47.833333 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:47.833190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:47.833464 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:47.833445    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:47.833548 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:47.833532    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret podName:0b046f46-d48b-4025-b9f9-37c5eea82ff9 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:51.833499072 +0000 UTC m=+10.025027268 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret") pod "global-pull-secret-syncer-hgkhl" (UID: "0b046f46-d48b-4025-b9f9-37c5eea82ff9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:47.833627 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:47.833613    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:47.833694 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:47.833658    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs podName:50de453c-4d08-4f49-9e7e-0d5e2221e72e nodeName:}" failed. No retries permitted until 2026-09-10 14:19:51.83364477 +0000 UTC m=+10.025172948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs") pod "network-metrics-daemon-qvxlq" (UID: "50de453c-4d08-4f49-9e7e-0d5e2221e72e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:47.934374 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:47.934309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqs5\" (UniqueName: \"kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5\") pod \"network-check-target-hn9m5\" (UID: \"dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33\") " pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:19:47.934578 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:47.934560    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:19:47.934662 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:47.934586    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:19:47.934662 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:47.934598    2580 projected.go:196] Error preparing data for projected volume kube-api-access-fjqs5 for pod openshift-network-diagnostics/network-check-target-hn9m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:47.934662 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:47.934657    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5 podName:dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:51.934638798 +0000 UTC m=+10.126166989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjqs5" (UniqueName: "kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5") pod "network-check-target-hn9m5" (UID: "dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:48.306926 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:48.306847    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:19:48.307395 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:48.306986    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:19:49.304821 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:49.304229    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:49.304821 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:49.304365    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:19:49.304821 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:49.304749    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:49.305148 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:49.304883    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:19:50.304190 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:50.304152    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:19:50.304587 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:50.304285    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:19:51.303565 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:51.303482    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:51.303565 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:51.303521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:51.308075 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:51.304013    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:19:51.308075 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:51.304616    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:19:51.869296 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:51.869258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs\") pod \"network-metrics-daemon-qvxlq\" (UID: \"50de453c-4d08-4f49-9e7e-0d5e2221e72e\") " pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:51.869480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:51.869313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:51.869480 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:51.869437    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:51.869480 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:51.869446    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:51.869591 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:51.869505    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret podName:0b046f46-d48b-4025-b9f9-37c5eea82ff9 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:59.869484326 +0000 UTC m=+18.061012502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret") pod "global-pull-secret-syncer-hgkhl" (UID: "0b046f46-d48b-4025-b9f9-37c5eea82ff9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:51.869591 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:51.869528    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs podName:50de453c-4d08-4f49-9e7e-0d5e2221e72e nodeName:}" failed. No retries permitted until 2026-09-10 14:19:59.869516518 +0000 UTC m=+18.061044698 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs") pod "network-metrics-daemon-qvxlq" (UID: "50de453c-4d08-4f49-9e7e-0d5e2221e72e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:51.970006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:51.969917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqs5\" (UniqueName: \"kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5\") pod \"network-check-target-hn9m5\" (UID: \"dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33\") " pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:19:51.970171 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:51.970124    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:19:51.970171 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:51.970150    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:19:51.970171 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:51.970164    2580 projected.go:196] Error preparing data for projected volume kube-api-access-fjqs5 for pod openshift-network-diagnostics/network-check-target-hn9m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:51.970336 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:51.970225    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5 podName:dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:59.970205844 +0000 UTC m=+18.161734040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjqs5" (UniqueName: "kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5") pod "network-check-target-hn9m5" (UID: "dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:19:52.305249 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:52.304777    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:19:52.305249 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:52.304884    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:19:53.304385 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:53.304351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:53.304841 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:53.304430    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:53.304841 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:53.304566    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:19:53.304841 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:53.304716    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:19:54.303899 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:54.303866    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:19:54.304106 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:54.304016    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:19:55.304648 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:55.304553    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:55.305236 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:55.304685    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:19:55.305236 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:55.304743    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:55.305236 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:55.304861    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:19:56.304577 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:56.304548    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:19:56.304760 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:56.304667    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:19:57.304036 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:57.303997    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:57.304036 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:57.304020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:57.304266 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:57.304134    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:19:57.304266 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:57.304219    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:19:58.304067 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:58.304036    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:19:58.304452 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:58.304129    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:19:59.303913 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:59.303871    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:59.303913 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:59.303906    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:59.304511 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:59.304021    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:19:59.304511 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:59.304138    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:19:59.926595 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:59.926562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:19:59.926793 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:19:59.926656    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs\") pod \"network-metrics-daemon-qvxlq\" (UID: \"50de453c-4d08-4f49-9e7e-0d5e2221e72e\") " pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:19:59.926793 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:59.926718    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:59.926793 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:59.926753    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:19:59.926793 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:59.926787    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret podName:0b046f46-d48b-4025-b9f9-37c5eea82ff9 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:15.926766845 +0000 UTC m=+34.118295038 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret") pod "global-pull-secret-syncer-hgkhl" (UID: "0b046f46-d48b-4025-b9f9-37c5eea82ff9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:19:59.927025 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:19:59.926806    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs podName:50de453c-4d08-4f49-9e7e-0d5e2221e72e nodeName:}" failed. No retries permitted until 2026-09-10 14:20:15.926797661 +0000 UTC m=+34.118325837 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs") pod "network-metrics-daemon-qvxlq" (UID: "50de453c-4d08-4f49-9e7e-0d5e2221e72e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:20:00.027416 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:00.027383    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqs5\" (UniqueName: \"kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5\") pod \"network-check-target-hn9m5\" (UID: \"dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33\") " pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:00.027587 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:00.027510    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:20:00.027587 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:00.027528    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:20:00.027587 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:00.027540    2580 projected.go:196] Error preparing data for projected volume kube-api-access-fjqs5 for pod openshift-network-diagnostics/network-check-target-hn9m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:20:00.027715 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:00.027599    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5 podName:dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:16.027584788 +0000 UTC m=+34.219112966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjqs5" (UniqueName: "kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5") pod "network-check-target-hn9m5" (UID: "dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:20:00.304280 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:00.304174    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:00.304689 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:00.304323    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:20:01.304285 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:01.304247    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:20:01.304285 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:01.304278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:20:01.304776 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:01.304379    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:20:01.304776 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:01.304538    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:20:02.304100 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:02.304081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:02.304183 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:02.304165    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:20:03.304461 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.304097    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:20:03.305161 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.304161    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:20:03.305161 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:03.304499    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:20:03.305161 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:03.304557    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:20:03.369167 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.369129    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vjqdq" event={"ID":"48aab852-7724-4969-b2c7-d6d2db10ec8d","Type":"ContainerStarted","Data":"2978d79986bde3850cd0a717e2f285eaeb9d2978d4320ac69c7d9bf47a8086a7"}
Sep 10 14:20:03.370616 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.370583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4bwvd" event={"ID":"3523580c-458a-486f-9cc6-8705298c0617","Type":"ContainerStarted","Data":"b656559595e8c83c8f748a39d1ce5605ff2652cabd2609a1cc88adf778c2ffb3"}
Sep 10 14:20:03.372024 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.372000    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n82cf" event={"ID":"496deff1-b6df-408f-8282-2b61f4563584","Type":"ContainerStarted","Data":"8cda342ffdf4a38e0e09d7925461f868e5c33f6d505fdcc1d151f513007a88c5"}
Sep 10 14:20:03.373320 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.373300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wwmw8" event={"ID":"2c791935-61bc-4e51-a9b6-fdc8076ee4c6","Type":"ContainerStarted","Data":"bd37991eeb2d34113fa97bcb2f0dd5b920b7cbf5fe76b7d75777dbb334c1a994"}
Sep 10 14:20:03.377293 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.377265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm" event={"ID":"56412239-e32d-418f-b321-34de77a92ce6","Type":"ContainerStarted","Data":"8349809502c9ca69e34f93be73166a9a26fdb38534642aed66be056ec3b24a52"}
Sep 10 14:20:03.378744 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.378721    2580 generic.go:356] "Generic (PLEG): container finished" podID="dab447291d2c7d1953740561c77a79c0" containerID="a1082f6680af4287cbd9080a745e4ac828857b0bf2062f53686fe15a19ca3a5b" exitCode=0
Sep 10 14:20:03.378823 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.378788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" event={"ID":"dab447291d2c7d1953740561c77a79c0","Type":"ContainerDied","Data":"a1082f6680af4287cbd9080a745e4ac828857b0bf2062f53686fe15a19ca3a5b"}
Sep 10 14:20:03.380303 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.380270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal" event={"ID":"e416b0faba17117313669ffa5d00f48b","Type":"ContainerStarted","Data":"719e685c41e152955f4071932938afa59ba59a81fd11b0a9f1a25c0877aa758f"}
Sep 10 14:20:03.383169 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.383150    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:20:03.383512 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.383493    2580 generic.go:356] "Generic (PLEG): container finished" podID="a7ef5dcd-b063-44f5-85ae-d9dc78521768" containerID="dc126fb45af9c8f772edb6671dc746998942ffee039aac04467bb8a13ee0b943" exitCode=1
Sep 10 14:20:03.383573 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.383552    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6" event={"ID":"a7ef5dcd-b063-44f5-85ae-d9dc78521768","Type":"ContainerStarted","Data":"c0cf51b5bae46bc7c127e7d0f461719f322adc3d4175b016851fb4028efab1dc"}
Sep 10 14:20:03.383621 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.383607    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6" event={"ID":"a7ef5dcd-b063-44f5-85ae-d9dc78521768","Type":"ContainerStarted","Data":"41dda6e84deed56c3a8bea47ed7164635129968b08b3703b04a1ae572d206355"}
Sep 10 14:20:03.383668 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.383645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6" event={"ID":"a7ef5dcd-b063-44f5-85ae-d9dc78521768","Type":"ContainerStarted","Data":"6fda054919cac7eae23c9201f24c32c42ec370b90be52db1f2ec4e39fe618a58"}
Sep 10 14:20:03.383730 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.383679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6" event={"ID":"a7ef5dcd-b063-44f5-85ae-d9dc78521768","Type":"ContainerStarted","Data":"91550c5d2b2452a05f8d4fdb9e6f4f771f9e445243cc556bb56ecb60beee13b4"}
Sep 10 14:20:03.383730 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.383712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6" event={"ID":"a7ef5dcd-b063-44f5-85ae-d9dc78521768","Type":"ContainerDied","Data":"dc126fb45af9c8f772edb6671dc746998942ffee039aac04467bb8a13ee0b943"}
Sep 10 14:20:03.383815 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.383754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6" event={"ID":"a7ef5dcd-b063-44f5-85ae-d9dc78521768","Type":"ContainerStarted","Data":"66bbbe6f2e7c28857d644107405cab781eb9cb262338938cf55fae6f460850c7"}
Sep 10 14:20:03.384962 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.384943    2580 generic.go:356] "Generic (PLEG): container finished" podID="51965344-8fbc-41c8-b3ef-2f553017edbe" containerID="d793cf11664e80dc3ee26345b2cd94d27b153b13dc723bebca0a1c9941a26eb6" exitCode=0
Sep 10 14:20:03.385055 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.385000    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf6ht" event={"ID":"51965344-8fbc-41c8-b3ef-2f553017edbe","Type":"ContainerDied","Data":"d793cf11664e80dc3ee26345b2cd94d27b153b13dc723bebca0a1c9941a26eb6"}
Sep 10 14:20:03.386587 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.386569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-47x2k" event={"ID":"20919d23-b9fc-4552-8082-4b83053f870b","Type":"ContainerStarted","Data":"b2c1463b85c6345a0e7e1be8e95265908d70fe66d06b695bea104bc0f723fbe1"}
Sep 10 14:20:03.399185 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.399130    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vjqdq" podStartSLOduration=3.555777826 podStartE2EDuration="21.399110564s" totalImagesPullingTime="17.843332738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:03.38296008 +0000 UTC m=+21.574488278" watchObservedRunningTime="2026-09-10 14:20:03.399110564 +0000 UTC m=+21.590638763"
Sep 10 14:20:03.412190 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.412118    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4bwvd" podStartSLOduration=7.700668339 podStartE2EDuration="21.412098062s" totalImagesPullingTime="13.711429723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:03.411939253 +0000 UTC m=+21.603467454" watchObservedRunningTime="2026-09-10 14:20:03.412098062 +0000 UTC m=+21.603626260"
Sep 10 14:20:03.425108 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.425058    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal" podStartSLOduration=20.425037968 podStartE2EDuration="20.425037968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:03.42434227 +0000 UTC m=+21.615870468" watchObservedRunningTime="2026-09-10 14:20:03.425037968 +0000 UTC m=+21.616566167"
Sep 10 14:20:03.439289 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.439245    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wwmw8" podStartSLOduration=3.521627595 podStartE2EDuration="21.439230824s" totalImagesPullingTime="17.917603229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:03.439091507 +0000 UTC m=+21.630619705" watchObservedRunningTime="2026-09-10 14:20:03.439230824 +0000 UTC m=+21.630759022"
Sep 10 14:20:03.452466 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.452404    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n82cf" podStartSLOduration=11.117316219 podStartE2EDuration="21.452384149s" totalImagesPullingTime="10.33506793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:03.451896354 +0000 UTC m=+21.643424551" watchObservedRunningTime="2026-09-10 14:20:03.452384149 +0000 UTC m=+21.643912346"
Sep 10 14:20:03.490450 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.490401    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-47x2k" podStartSLOduration=3.510163401 podStartE2EDuration="21.490384088s" totalImagesPullingTime="17.980220687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:03.489697965 +0000 UTC m=+21.681226164" watchObservedRunningTime="2026-09-10 14:20:03.490384088 +0000 UTC m=+21.681912283"
Sep 10 14:20:03.627465 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.627373    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n82cf_496deff1-b6df-408f-8282-2b61f4563584/konnectivity-agent/0.log" limit=1024
Sep 10 14:20:03.710262 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.710228    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n82cf_496deff1-b6df-408f-8282-2b61f4563584/konnectivity-agent/0.log" limit=1024
Sep 10 14:20:03.958361 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:03.958338    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 14:20:04.254895 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:04.254731    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T14:20:03.958355887Z","UUID":"3403aea3-ad8c-4d5f-b6cb-632f495c2eb9","Handler":null,"Name":"","Endpoint":""}
Sep 10 14:20:04.256440 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:04.256417    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 14:20:04.256583 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:04.256460    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 14:20:04.303904 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:04.303873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:04.304112 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:04.304018    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:20:04.391627 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:04.391591    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm" event={"ID":"56412239-e32d-418f-b321-34de77a92ce6","Type":"ContainerStarted","Data":"7b6ebc84ddf40fafb3883fb125e2cea8aef2f7a5a8d4dae3aeb21ca12d64240e"}
Sep 10 14:20:04.394081 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:04.394040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" event={"ID":"dab447291d2c7d1953740561c77a79c0","Type":"ContainerStarted","Data":"e385bd9d71950ae5d3993e0c78c955f22f505fcfbe632677ed6465211949b4cf"}
Sep 10 14:20:04.395734 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:04.395704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mtztw" event={"ID":"5f4523e7-b3bf-40bb-90f5-40b2a7b26ce3","Type":"ContainerStarted","Data":"56bd83d2e8cd93c18642872b91308871bf0ad3be32de1b42f4f3d0983fbe369c"}
Sep 10 14:20:04.424848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:04.424792    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mtztw" podStartSLOduration=8.733757657 podStartE2EDuration="22.424775742s" totalImagesPullingTime="13.691018085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:04.424708867 +0000 UTC m=+22.616237066" watchObservedRunningTime="2026-09-10 14:20:04.424775742 +0000 UTC m=+22.616303940"
Sep 10 14:20:04.425268 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:04.425235    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" podStartSLOduration=21.425225545 podStartE2EDuration="21.425225545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:04.411540873 +0000 UTC m=+22.603069072" watchObservedRunningTime="2026-09-10 14:20:04.425225545 +0000 UTC m=+22.616753743"
Sep 10 14:20:04.543238 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:04.543143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n82cf_496deff1-b6df-408f-8282-2b61f4563584/konnectivity-agent/0.log" limit=1024
Sep 10 14:20:04.855812 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:04.855732    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n82cf_496deff1-b6df-408f-8282-2b61f4563584/konnectivity-agent/0.log" limit=1024
Sep 10 14:20:05.304506 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:05.304310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:20:05.304621 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:05.304353    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:20:05.304621 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:05.304610    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:20:05.304736 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:05.304698    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:20:05.400449 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:05.400268    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm" event={"ID":"56412239-e32d-418f-b321-34de77a92ce6","Type":"ContainerStarted","Data":"73371bd45ed638a48d9f46d536019ed5834c7a4b73de132ff371ad2a710236eb"}
Sep 10 14:20:05.403513 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:05.403492    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:20:05.404130 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:05.404104    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6" event={"ID":"a7ef5dcd-b063-44f5-85ae-d9dc78521768","Type":"ContainerStarted","Data":"0279e4c5cfcb5a042cb39413e9d3cb7c2fc2755ddd0b12fe6a45bd903733e842"}
Sep 10 14:20:05.416447 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:05.416405    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zwm" podStartSLOduration=3.235915579 podStartE2EDuration="23.416391043s" totalImagesPullingTime="20.180475464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:05.41628819 +0000 UTC m=+23.607816392" watchObservedRunningTime="2026-09-10 14:20:05.416391043 +0000 UTC m=+23.607919237"
Sep 10 14:20:06.304116 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:06.304067    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:06.304402 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:06.304241    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:20:06.869110 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:06.869071    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n82cf"
Sep 10 14:20:06.869916 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:06.869896    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n82cf"
Sep 10 14:20:07.303775 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:07.303745    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:20:07.303952 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:07.303753    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:20:07.303952 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:07.303877    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:20:07.304099 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:07.303985    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:20:07.409353 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:07.409324    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n82cf"
Sep 10 14:20:07.409880 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:07.409860    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n82cf"
Sep 10 14:20:08.303716 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:08.303546    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:08.304223 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:08.303821    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:20:08.414431 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:08.414399    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:20:08.414804 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:08.414783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6" event={"ID":"a7ef5dcd-b063-44f5-85ae-d9dc78521768","Type":"ContainerStarted","Data":"0444cf8ed04d510d6d442e1c2cdaa7b3ab13a9ec4919ae05044b0c3c9e2451ee"}
Sep 10 14:20:08.415148 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:08.415113    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:20:08.415256 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:08.415208    2580 scope.go:117] "RemoveContainer" containerID="dc126fb45af9c8f772edb6671dc746998942ffee039aac04467bb8a13ee0b943"
Sep 10 14:20:08.417246 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:08.417221    2580 generic.go:356] "Generic (PLEG): container finished" podID="51965344-8fbc-41c8-b3ef-2f553017edbe" containerID="41c925676beda941efeac7beddcd7cc7d5650729031d5b263bd7facade050893" exitCode=0
Sep 10 14:20:08.417345 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:08.417312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf6ht" event={"ID":"51965344-8fbc-41c8-b3ef-2f553017edbe","Type":"ContainerDied","Data":"41c925676beda941efeac7beddcd7cc7d5650729031d5b263bd7facade050893"}
Sep 10 14:20:08.431337 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:08.431318    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:20:09.289595 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:09.289562    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvxlq"]
Sep 10 14:20:09.289738 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:09.289708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:20:09.289824 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:09.289800    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:20:09.292446 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:09.292371    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hn9m5"]
Sep 10 14:20:09.292567 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:09.292473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:09.292567 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:09.292556    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:20:09.302913 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:09.302853    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hgkhl"]
Sep 10 14:20:09.303030 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:09.302934    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:20:09.303030 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:09.303021    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:20:09.423631 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:09.423604    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:20:09.424450 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:09.423965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6" event={"ID":"a7ef5dcd-b063-44f5-85ae-d9dc78521768","Type":"ContainerStarted","Data":"de816ea052c7707cca60f9fa72080f05ed8e9f2e617bb0260e0a403f764b25d1"}
Sep 10 14:20:09.424450 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:09.424060    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 14:20:09.424450 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:09.424344    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:20:09.426371 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:09.426350    2580 generic.go:356] "Generic (PLEG): container finished" podID="51965344-8fbc-41c8-b3ef-2f553017edbe" containerID="8197d4a37ab4cced5ac62b9186fa106ee15a8a3b4fe9e85f461ac0cde98ef598" exitCode=0
Sep 10 14:20:09.426457 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:09.426434    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf6ht" event={"ID":"51965344-8fbc-41c8-b3ef-2f553017edbe","Type":"ContainerDied","Data":"8197d4a37ab4cced5ac62b9186fa106ee15a8a3b4fe9e85f461ac0cde98ef598"}
Sep 10 14:20:09.439962 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:09.439941    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:20:09.449865 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:09.449825    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6" podStartSLOduration=9.535160497 podStartE2EDuration="27.449810275s" totalImagesPullingTime="17.914649778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:09.448169234 +0000 UTC m=+27.639697452" watchObservedRunningTime="2026-09-10 14:20:09.449810275 +0000 UTC m=+27.641338474"
Sep 10 14:20:09.529230 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:09.529196    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:20:10.431502 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:10.431405    2580 generic.go:356] "Generic (PLEG): container finished" podID="51965344-8fbc-41c8-b3ef-2f553017edbe" containerID="3c1570d85ec9d30ed9acb7e2e635783262f45edb1b1f2c9bfdd70630aca05c80" exitCode=0
Sep 10 14:20:10.431903 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:10.431493    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf6ht" event={"ID":"51965344-8fbc-41c8-b3ef-2f553017edbe","Type":"ContainerDied","Data":"3c1570d85ec9d30ed9acb7e2e635783262f45edb1b1f2c9bfdd70630aca05c80"}
Sep 10 14:20:11.304134 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:11.303934    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:11.304304 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:11.303939    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:20:11.304304 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:11.304218    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:20:11.304424 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:11.304314    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:20:11.304424 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:11.303985    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:20:11.304524 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:11.304471    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:20:13.304221 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:13.304191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:20:13.304221 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:13.304220    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:20:13.304743 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:13.304380    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:20:13.304743 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:13.304410    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:13.304743 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:13.304500    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:20:13.304743 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:13.304605    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:20:15.304519 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.304446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:20:15.304922 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.304446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:15.304922 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:15.304543    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkhl" podUID="0b046f46-d48b-4025-b9f9-37c5eea82ff9"
Sep 10 14:20:15.304922 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:15.304609    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hn9m5" podUID="dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33"
Sep 10 14:20:15.304922 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.304445    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:20:15.304922 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:15.304677    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxlq" podUID="50de453c-4d08-4f49-9e7e-0d5e2221e72e"
Sep 10 14:20:15.446182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.446150    2580 generic.go:356] "Generic (PLEG): container finished" podID="51965344-8fbc-41c8-b3ef-2f553017edbe" containerID="1e2a596df7290d9f98ae5d19afafae42ea59646bee413a94fb5b372ee1f09a32" exitCode=0
Sep 10 14:20:15.446336 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.446196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf6ht" event={"ID":"51965344-8fbc-41c8-b3ef-2f553017edbe","Type":"ContainerDied","Data":"1e2a596df7290d9f98ae5d19afafae42ea59646bee413a94fb5b372ee1f09a32"}
Sep 10 14:20:15.577297 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.577276    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeReady"
Sep 10 14:20:15.577414 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.577403    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 14:20:15.627965 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.627912    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ss76l"]
Sep 10 14:20:15.642050 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.642018    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-546fb9cb8f-xckpj"]
Sep 10 14:20:15.642213 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.642149    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:15.644668 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.644599    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bskzl\""
Sep 10 14:20:15.645532 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.645512    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 14:20:15.645776 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.645605    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 14:20:15.658064 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.658040    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6z7bn"]
Sep 10 14:20:15.658156 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.658148    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.660476 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.660456    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 14:20:15.660583 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.660479    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 14:20:15.660583 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.660501    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cgxdb\""
Sep 10 14:20:15.660583 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.660506    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 14:20:15.664987 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.664953    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 14:20:15.672255 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.672239    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6z7bn"]
Sep 10 14:20:15.672349 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.672263    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ss76l"]
Sep 10 14:20:15.672349 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.672282    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-546fb9cb8f-xckpj"]
Sep 10 14:20:15.672427 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.672361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6z7bn"
Sep 10 14:20:15.674815 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.674775    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 14:20:15.674882 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.674829    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tsxlz\""
Sep 10 14:20:15.674933 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.674911    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 14:20:15.675070 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.675052    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 14:20:15.739202 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.739173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-bound-sa-token\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.739202 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.739205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhqxl\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-kube-api-access-mhqxl\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.739408 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.739226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:15.739408 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.739241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8bdbb9e-6d22-4d3e-af97-c34159145a29-tmp-dir\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:15.739408 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.739255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.739408 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.739361    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f94d471e-af4c-46f6-b086-1ffd11503f9b-image-registry-private-configuration\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.739408 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.739390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-certificates\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.739408 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.739408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f94d471e-af4c-46f6-b086-1ffd11503f9b-trusted-ca\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.739607 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.739448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e8bdbb9e-6d22-4d3e-af97-c34159145a29-config-volume\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:15.739607 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.739462    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f94d471e-af4c-46f6-b086-1ffd11503f9b-installation-pull-secrets\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.739607 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.739487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfxwd\" (UniqueName: \"kubernetes.io/projected/e8bdbb9e-6d22-4d3e-af97-c34159145a29-kube-api-access-gfxwd\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:15.739607 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.739515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f94d471e-af4c-46f6-b086-1ffd11503f9b-ca-trust-extracted\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.840528 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.840458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfxwd\" (UniqueName: \"kubernetes.io/projected/e8bdbb9e-6d22-4d3e-af97-c34159145a29-kube-api-access-gfxwd\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:15.840528 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.840494    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f94d471e-af4c-46f6-b086-1ffd11503f9b-ca-trust-extracted\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.840528 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.840515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd2cae1f-0990-4824-857c-a1872793416d-cert\") pod \"ingress-canary-6z7bn\" (UID: \"bd2cae1f-0990-4824-857c-a1872793416d\") " pod="openshift-ingress-canary/ingress-canary-6z7bn"
Sep 10 14:20:15.840766 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.840557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-bound-sa-token\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.840766 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.840573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhqxl\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-kube-api-access-mhqxl\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.840766 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.840589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:15.840766 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.840609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8bdbb9e-6d22-4d3e-af97-c34159145a29-tmp-dir\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:15.840766 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.840632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.840766 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:15.840706    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:20:15.840766 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:15.840720    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546fb9cb8f-xckpj: secret "image-registry-tls" not found
Sep 10 14:20:15.840766 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:15.840729    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:20:15.840766 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.840703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvm49\" (UniqueName: \"kubernetes.io/projected/bd2cae1f-0990-4824-857c-a1872793416d-kube-api-access-gvm49\") pod \"ingress-canary-6z7bn\" (UID: \"bd2cae1f-0990-4824-857c-a1872793416d\") " pod="openshift-ingress-canary/ingress-canary-6z7bn"
Sep 10 14:20:15.841176 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.840795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f94d471e-af4c-46f6-b086-1ffd11503f9b-image-registry-private-configuration\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.841176 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:15.840822    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls podName:e8bdbb9e-6d22-4d3e-af97-c34159145a29 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:16.340801633 +0000 UTC m=+34.532329827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls") pod "dns-default-ss76l" (UID: "e8bdbb9e-6d22-4d3e-af97-c34159145a29") : secret "dns-default-metrics-tls" not found
Sep 10 14:20:15.841176 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:15.840862    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls podName:f94d471e-af4c-46f6-b086-1ffd11503f9b nodeName:}" failed. No retries permitted until 2026-09-10 14:20:16.340852013 +0000 UTC m=+34.532380189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls") pod "image-registry-546fb9cb8f-xckpj" (UID: "f94d471e-af4c-46f6-b086-1ffd11503f9b") : secret "image-registry-tls" not found
Sep 10 14:20:15.841176 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.840880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-certificates\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.841176 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.840913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f94d471e-af4c-46f6-b086-1ffd11503f9b-trusted-ca\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.841176 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.840922    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f94d471e-af4c-46f6-b086-1ffd11503f9b-ca-trust-extracted\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.841176 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.840998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e8bdbb9e-6d22-4d3e-af97-c34159145a29-config-volume\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:15.841176 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.841031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f94d471e-af4c-46f6-b086-1ffd11503f9b-installation-pull-secrets\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.841524 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.841473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-certificates\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.841574 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.841532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e8bdbb9e-6d22-4d3e-af97-c34159145a29-config-volume\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:15.841799 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.841780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f94d471e-af4c-46f6-b086-1ffd11503f9b-trusted-ca\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.844646 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.844497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8bdbb9e-6d22-4d3e-af97-c34159145a29-tmp-dir\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:15.844746 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.844657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f94d471e-af4c-46f6-b086-1ffd11503f9b-installation-pull-secrets\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.844746 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.844669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f94d471e-af4c-46f6-b086-1ffd11503f9b-image-registry-private-configuration\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.850668 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.850638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfxwd\" (UniqueName: \"kubernetes.io/projected/e8bdbb9e-6d22-4d3e-af97-c34159145a29-kube-api-access-gfxwd\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:15.851559 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.851540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-bound-sa-token\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.851741 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.851724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhqxl\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-kube-api-access-mhqxl\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:15.942084 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.942052    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd2cae1f-0990-4824-857c-a1872793416d-cert\") pod \"ingress-canary-6z7bn\" (UID: \"bd2cae1f-0990-4824-857c-a1872793416d\") " pod="openshift-ingress-canary/ingress-canary-6z7bn"
Sep 10 14:20:15.942278 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.942122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs\") pod \"network-metrics-daemon-qvxlq\" (UID: \"50de453c-4d08-4f49-9e7e-0d5e2221e72e\") " pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:20:15.942278 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.942139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvm49\" (UniqueName: \"kubernetes.io/projected/bd2cae1f-0990-4824-857c-a1872793416d-kube-api-access-gvm49\") pod \"ingress-canary-6z7bn\" (UID: \"bd2cae1f-0990-4824-857c-a1872793416d\") " pod="openshift-ingress-canary/ingress-canary-6z7bn"
Sep 10 14:20:15.942278 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.942158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:20:15.942278 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:15.942233    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:20:15.942278 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:15.942257    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:20:15.942486 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:15.942308    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret podName:0b046f46-d48b-4025-b9f9-37c5eea82ff9 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:47.942295416 +0000 UTC m=+66.133823592 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret") pod "global-pull-secret-syncer-hgkhl" (UID: "0b046f46-d48b-4025-b9f9-37c5eea82ff9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:20:15.942486 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:15.942322    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs podName:50de453c-4d08-4f49-9e7e-0d5e2221e72e nodeName:}" failed. No retries permitted until 2026-09-10 14:20:47.942316765 +0000 UTC m=+66.133844941 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs") pod "network-metrics-daemon-qvxlq" (UID: "50de453c-4d08-4f49-9e7e-0d5e2221e72e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:20:15.944540 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.944510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd2cae1f-0990-4824-857c-a1872793416d-cert\") pod \"ingress-canary-6z7bn\" (UID: \"bd2cae1f-0990-4824-857c-a1872793416d\") " pod="openshift-ingress-canary/ingress-canary-6z7bn"
Sep 10 14:20:15.951175 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.951153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvm49\" (UniqueName: \"kubernetes.io/projected/bd2cae1f-0990-4824-857c-a1872793416d-kube-api-access-gvm49\") pod \"ingress-canary-6z7bn\" (UID: \"bd2cae1f-0990-4824-857c-a1872793416d\") " pod="openshift-ingress-canary/ingress-canary-6z7bn"
Sep 10 14:20:15.988854 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:15.988820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6z7bn"
Sep 10 14:20:16.042933 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:16.042901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqs5\" (UniqueName: \"kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5\") pod \"network-check-target-hn9m5\" (UID: \"dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33\") " pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:16.043134 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:16.043110    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:20:16.043176 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:16.043145    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:20:16.043176 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:16.043159    2580 projected.go:196] Error preparing data for projected volume kube-api-access-fjqs5 for pod openshift-network-diagnostics/network-check-target-hn9m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:20:16.043243 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:16.043224    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5 podName:dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:48.043206143 +0000 UTC m=+66.234734336 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjqs5" (UniqueName: "kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5") pod "network-check-target-hn9m5" (UID: "dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:20:16.286451 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:16.286359    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6z7bn"]
Sep 10 14:20:16.290165 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:20:16.290142    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2cae1f_0990_4824_857c_a1872793416d.slice/crio-59bbb28a7210545b0b4051b048eb664c2ac97ce775e5c98fec251f1f13e45489 WatchSource:0}: Error finding container 59bbb28a7210545b0b4051b048eb664c2ac97ce775e5c98fec251f1f13e45489: Status 404 returned error can't find the container with id 59bbb28a7210545b0b4051b048eb664c2ac97ce775e5c98fec251f1f13e45489
Sep 10 14:20:16.345195 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:16.345157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:16.345195 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:16.345195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:16.345871 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:16.345310    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:20:16.345871 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:16.345324    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546fb9cb8f-xckpj: secret "image-registry-tls" not found
Sep 10 14:20:16.345871 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:16.345309    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:20:16.345871 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:16.345376    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls podName:f94d471e-af4c-46f6-b086-1ffd11503f9b nodeName:}" failed. No retries permitted until 2026-09-10 14:20:17.345359432 +0000 UTC m=+35.536887624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls") pod "image-registry-546fb9cb8f-xckpj" (UID: "f94d471e-af4c-46f6-b086-1ffd11503f9b") : secret "image-registry-tls" not found
Sep 10 14:20:16.345871 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:16.345404    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls podName:e8bdbb9e-6d22-4d3e-af97-c34159145a29 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:17.345392111 +0000 UTC m=+35.536920288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls") pod "dns-default-ss76l" (UID: "e8bdbb9e-6d22-4d3e-af97-c34159145a29") : secret "dns-default-metrics-tls" not found
Sep 10 14:20:16.451218 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:16.451187    2580 generic.go:356] "Generic (PLEG): container finished" podID="51965344-8fbc-41c8-b3ef-2f553017edbe" containerID="da73f2e6a8d389e3b46b0f470b2fe1f699411991f10c60bd0411acdbede049d5" exitCode=0
Sep 10 14:20:16.451431 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:16.451265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf6ht" event={"ID":"51965344-8fbc-41c8-b3ef-2f553017edbe","Type":"ContainerDied","Data":"da73f2e6a8d389e3b46b0f470b2fe1f699411991f10c60bd0411acdbede049d5"}
Sep 10 14:20:16.452466 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:16.452444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6z7bn" event={"ID":"bd2cae1f-0990-4824-857c-a1872793416d","Type":"ContainerStarted","Data":"59bbb28a7210545b0b4051b048eb664c2ac97ce775e5c98fec251f1f13e45489"}
Sep 10 14:20:17.304322 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:17.304289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:20:17.304614 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:17.304299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:20:17.304614 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:17.304437    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:17.308118 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:17.308091    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 14:20:17.308262 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:17.308185    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 14:20:17.308262 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:17.308188    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 14:20:17.308387 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:17.308310    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q47dh\""
Sep 10 14:20:17.308387 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:17.308346    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c5vvv\""
Sep 10 14:20:17.308486 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:17.308435    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 14:20:17.353490 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:17.353457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:17.354225 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:17.353494    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:17.354225 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:17.353596    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:20:17.354225 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:17.353669    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls podName:e8bdbb9e-6d22-4d3e-af97-c34159145a29 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:19.353648641 +0000 UTC m=+37.545176823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls") pod "dns-default-ss76l" (UID: "e8bdbb9e-6d22-4d3e-af97-c34159145a29") : secret "dns-default-metrics-tls" not found
Sep 10 14:20:17.354225 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:17.353602    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:20:17.354225 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:17.353700    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546fb9cb8f-xckpj: secret "image-registry-tls" not found
Sep 10 14:20:17.354225 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:17.353749    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls podName:f94d471e-af4c-46f6-b086-1ffd11503f9b nodeName:}" failed. No retries permitted until 2026-09-10 14:20:19.353736309 +0000 UTC m=+37.545264485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls") pod "image-registry-546fb9cb8f-xckpj" (UID: "f94d471e-af4c-46f6-b086-1ffd11503f9b") : secret "image-registry-tls" not found
Sep 10 14:20:17.459173 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:17.459133    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nf6ht" event={"ID":"51965344-8fbc-41c8-b3ef-2f553017edbe","Type":"ContainerStarted","Data":"594daf9161c0fecaec0b9cc702edd38be4ce791afa598d1d2b4fced45cd50153"}
Sep 10 14:20:17.479627 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:17.479576    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nf6ht" podStartSLOduration=8.157329861 podStartE2EDuration="35.479560269s" totalImagesPullingTime="27.322230408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:17.478412822 +0000 UTC m=+35.669941022" watchObservedRunningTime="2026-09-10 14:20:17.479560269 +0000 UTC m=+35.671088467"
Sep 10 14:20:19.370451 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:19.370270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:19.370451 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:19.370434    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:20:19.370451 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:19.370457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:19.370872 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:19.370499    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls podName:e8bdbb9e-6d22-4d3e-af97-c34159145a29 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:23.370484576 +0000 UTC m=+41.562012757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls") pod "dns-default-ss76l" (UID: "e8bdbb9e-6d22-4d3e-af97-c34159145a29") : secret "dns-default-metrics-tls" not found
Sep 10 14:20:19.370872 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:19.370541    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:20:19.370872 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:19.370552    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546fb9cb8f-xckpj: secret "image-registry-tls" not found
Sep 10 14:20:19.370872 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:19.370591    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls podName:f94d471e-af4c-46f6-b086-1ffd11503f9b nodeName:}" failed. No retries permitted until 2026-09-10 14:20:23.370581283 +0000 UTC m=+41.562109459 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls") pod "image-registry-546fb9cb8f-xckpj" (UID: "f94d471e-af4c-46f6-b086-1ffd11503f9b") : secret "image-registry-tls" not found
Sep 10 14:20:19.465026 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:19.464996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6z7bn" event={"ID":"bd2cae1f-0990-4824-857c-a1872793416d","Type":"ContainerStarted","Data":"b990fb3839bd21b62530d05880805a0505c36ec9a6904de3606b1a2c5022754b"}
Sep 10 14:20:19.479498 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:19.479455    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6z7bn" podStartSLOduration=1.9161582369999999 podStartE2EDuration="4.479438668s" totalImagesPullingTime="2.563280431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:19.478880091 +0000 UTC m=+37.670408290" watchObservedRunningTime="2026-09-10 14:20:19.479438668 +0000 UTC m=+37.670966866"
Sep 10 14:20:21.041151 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.041122    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wcprc"]
Sep 10 14:20:21.055746 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.055722    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.058279 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.058230    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 14:20:21.058415 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.058318    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 14:20:21.058482 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.058414    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 14:20:21.058529 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.058490    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mdxkp\""
Sep 10 14:20:21.059295 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.059274    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 14:20:21.060232 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.060212    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wcprc"]
Sep 10 14:20:21.183192 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.183152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2b0aadca-c96f-4125-addf-c701034d2576-crio-socket\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.183192 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.183191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b0aadca-c96f-4125-addf-c701034d2576-data-volume\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.183386 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.183304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.183386 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.183334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.183386 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.183363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8pz\" (UniqueName: \"kubernetes.io/projected/2b0aadca-c96f-4125-addf-c701034d2576-kube-api-access-ws8pz\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.284104 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.284063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b0aadca-c96f-4125-addf-c701034d2576-data-volume\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.284270 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.284159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.284270 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.284182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.284270 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.284208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8pz\" (UniqueName: \"kubernetes.io/projected/2b0aadca-c96f-4125-addf-c701034d2576-kube-api-access-ws8pz\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.284270 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.284257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2b0aadca-c96f-4125-addf-c701034d2576-crio-socket\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.284464 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:21.284293    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:20:21.284464 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:21.284346    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:20:21.284464 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:21.284360    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm podName:2b0aadca-c96f-4125-addf-c701034d2576 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:21.784343638 +0000 UTC m=+39.975871828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wcprc" (UID: "2b0aadca-c96f-4125-addf-c701034d2576") : configmap "kube-rbac-proxy" not found
Sep 10 14:20:21.284464 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.284389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b0aadca-c96f-4125-addf-c701034d2576-data-volume\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.284464 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:21.284400    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls podName:2b0aadca-c96f-4125-addf-c701034d2576 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:21.784385471 +0000 UTC m=+39.975913646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcprc" (UID: "2b0aadca-c96f-4125-addf-c701034d2576") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:20:21.284700 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.284488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2b0aadca-c96f-4125-addf-c701034d2576-crio-socket\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.292941 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.292891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8pz\" (UniqueName: \"kubernetes.io/projected/2b0aadca-c96f-4125-addf-c701034d2576-kube-api-access-ws8pz\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.789826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.789734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.789826 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:21.789774    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:21.790053 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:21.789879    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:20:21.790053 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:21.789897    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:20:21.790053 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:21.789945    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm podName:2b0aadca-c96f-4125-addf-c701034d2576 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:22.789928979 +0000 UTC m=+40.981457154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wcprc" (UID: "2b0aadca-c96f-4125-addf-c701034d2576") : configmap "kube-rbac-proxy" not found
Sep 10 14:20:21.790053 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:21.789959    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls podName:2b0aadca-c96f-4125-addf-c701034d2576 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:22.789953857 +0000 UTC m=+40.981482034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcprc" (UID: "2b0aadca-c96f-4125-addf-c701034d2576") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:20:22.798258 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:22.798226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:22.798258 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:22.798264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:22.798642 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:22.798353    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:20:22.798642 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:22.798421    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm podName:2b0aadca-c96f-4125-addf-c701034d2576 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:24.798402518 +0000 UTC m=+42.989930710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wcprc" (UID: "2b0aadca-c96f-4125-addf-c701034d2576") : configmap "kube-rbac-proxy" not found
Sep 10 14:20:22.798642 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:22.798423    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:20:22.798642 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:22.798457    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls podName:2b0aadca-c96f-4125-addf-c701034d2576 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:24.798447463 +0000 UTC m=+42.989975639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcprc" (UID: "2b0aadca-c96f-4125-addf-c701034d2576") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:20:23.404170 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:23.404141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:23.404335 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:23.404180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:23.404335 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:23.404294    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:20:23.404405 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:23.404343    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls podName:e8bdbb9e-6d22-4d3e-af97-c34159145a29 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:31.404330148 +0000 UTC m=+49.595858330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls") pod "dns-default-ss76l" (UID: "e8bdbb9e-6d22-4d3e-af97-c34159145a29") : secret "dns-default-metrics-tls" not found
Sep 10 14:20:23.404405 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:23.404298    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:20:23.404405 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:23.404386    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546fb9cb8f-xckpj: secret "image-registry-tls" not found
Sep 10 14:20:23.404516 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:23.404427    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls podName:f94d471e-af4c-46f6-b086-1ffd11503f9b nodeName:}" failed. No retries permitted until 2026-09-10 14:20:31.404416029 +0000 UTC m=+49.595944204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls") pod "image-registry-546fb9cb8f-xckpj" (UID: "f94d471e-af4c-46f6-b086-1ffd11503f9b") : secret "image-registry-tls" not found
Sep 10 14:20:23.993401 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:23.993375    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vjqdq_48aab852-7724-4969-b2c7-d6d2db10ec8d/dns-node-resolver/0.log"
Sep 10 14:20:24.139138 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.139106    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-4ppd9"]
Sep 10 14:20:24.159236 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.159203    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-4ppd9"]
Sep 10 14:20:24.159373 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.159290    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-4ppd9"
Sep 10 14:20:24.161652 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.161626    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 14:20:24.161856 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.161696    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 14:20:24.161856 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.161725    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 14:20:24.162876 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.162857    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-m45v6\""
Sep 10 14:20:24.162930 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.162857    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 14:20:24.312420 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.312345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0c544b99-f864-4d9f-8721-27f2ec0f9f4b-signing-cabundle\") pod \"service-ca-75d99c484-4ppd9\" (UID: \"0c544b99-f864-4d9f-8721-27f2ec0f9f4b\") " pod="openshift-service-ca/service-ca-75d99c484-4ppd9"
Sep 10 14:20:24.312420 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.312390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czpwx\" (UniqueName: \"kubernetes.io/projected/0c544b99-f864-4d9f-8721-27f2ec0f9f4b-kube-api-access-czpwx\") pod \"service-ca-75d99c484-4ppd9\" (UID: \"0c544b99-f864-4d9f-8721-27f2ec0f9f4b\") " pod="openshift-service-ca/service-ca-75d99c484-4ppd9"
Sep 10 14:20:24.312420 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.312407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0c544b99-f864-4d9f-8721-27f2ec0f9f4b-signing-key\") pod \"service-ca-75d99c484-4ppd9\" (UID: \"0c544b99-f864-4d9f-8721-27f2ec0f9f4b\") " pod="openshift-service-ca/service-ca-75d99c484-4ppd9"
Sep 10 14:20:24.413032 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.413001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czpwx\" (UniqueName: \"kubernetes.io/projected/0c544b99-f864-4d9f-8721-27f2ec0f9f4b-kube-api-access-czpwx\") pod \"service-ca-75d99c484-4ppd9\" (UID: \"0c544b99-f864-4d9f-8721-27f2ec0f9f4b\") " pod="openshift-service-ca/service-ca-75d99c484-4ppd9"
Sep 10 14:20:24.413032 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.413034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0c544b99-f864-4d9f-8721-27f2ec0f9f4b-signing-key\") pod \"service-ca-75d99c484-4ppd9\" (UID: \"0c544b99-f864-4d9f-8721-27f2ec0f9f4b\") " pod="openshift-service-ca/service-ca-75d99c484-4ppd9"
Sep 10 14:20:24.413262 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.413116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0c544b99-f864-4d9f-8721-27f2ec0f9f4b-signing-cabundle\") pod \"service-ca-75d99c484-4ppd9\" (UID: \"0c544b99-f864-4d9f-8721-27f2ec0f9f4b\") " pod="openshift-service-ca/service-ca-75d99c484-4ppd9"
Sep 10 14:20:24.413776 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.413759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0c544b99-f864-4d9f-8721-27f2ec0f9f4b-signing-cabundle\") pod \"service-ca-75d99c484-4ppd9\" (UID: \"0c544b99-f864-4d9f-8721-27f2ec0f9f4b\") " pod="openshift-service-ca/service-ca-75d99c484-4ppd9"
Sep 10 14:20:24.415432 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.415404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0c544b99-f864-4d9f-8721-27f2ec0f9f4b-signing-key\") pod \"service-ca-75d99c484-4ppd9\" (UID: \"0c544b99-f864-4d9f-8721-27f2ec0f9f4b\") " pod="openshift-service-ca/service-ca-75d99c484-4ppd9"
Sep 10 14:20:24.420928 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.420902    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czpwx\" (UniqueName: \"kubernetes.io/projected/0c544b99-f864-4d9f-8721-27f2ec0f9f4b-kube-api-access-czpwx\") pod \"service-ca-75d99c484-4ppd9\" (UID: \"0c544b99-f864-4d9f-8721-27f2ec0f9f4b\") " pod="openshift-service-ca/service-ca-75d99c484-4ppd9"
Sep 10 14:20:24.467872 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.467839    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-4ppd9"
Sep 10 14:20:24.593254 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.593198    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4bwvd_3523580c-458a-486f-9cc6-8705298c0617/node-ca/0.log"
Sep 10 14:20:24.754210 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.754180    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-4ppd9"]
Sep 10 14:20:24.757651 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:20:24.757628    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c544b99_f864_4d9f_8721_27f2ec0f9f4b.slice/crio-80594a8f838ff35c703b2cfcc179d10d75f6016c6e54862e3f79f7b7a542f235 WatchSource:0}: Error finding container 80594a8f838ff35c703b2cfcc179d10d75f6016c6e54862e3f79f7b7a542f235: Status 404 returned error can't find the container with id 80594a8f838ff35c703b2cfcc179d10d75f6016c6e54862e3f79f7b7a542f235
Sep 10 14:20:24.815765 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.815731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:24.815765 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:24.815768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:24.815952 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:24.815850    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:20:24.815952 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:24.815892    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:20:24.815952 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:24.815918    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm podName:2b0aadca-c96f-4125-addf-c701034d2576 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:28.81589687 +0000 UTC m=+47.007425046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wcprc" (UID: "2b0aadca-c96f-4125-addf-c701034d2576") : configmap "kube-rbac-proxy" not found
Sep 10 14:20:24.815952 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:24.815952    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls podName:2b0aadca-c96f-4125-addf-c701034d2576 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:28.815938751 +0000 UTC m=+47.007466927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcprc" (UID: "2b0aadca-c96f-4125-addf-c701034d2576") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:20:25.394254 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:25.394225    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6z7bn_bd2cae1f-0990-4824-857c-a1872793416d/serve-healthcheck-canary/0.log"
Sep 10 14:20:25.479496 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:25.479454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-4ppd9" event={"ID":"0c544b99-f864-4d9f-8721-27f2ec0f9f4b","Type":"ContainerStarted","Data":"80594a8f838ff35c703b2cfcc179d10d75f6016c6e54862e3f79f7b7a542f235"}
Sep 10 14:20:27.486059 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:27.485962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-4ppd9" event={"ID":"0c544b99-f864-4d9f-8721-27f2ec0f9f4b","Type":"ContainerStarted","Data":"a6a187936fa49d1b29103232215573b1f8a335046d29fb2ed830a4862834b3b9"}
Sep 10 14:20:27.504521 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:27.504465    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-4ppd9" podStartSLOduration=1.09215592 podStartE2EDuration="3.504448865s" totalImagesPullingTime="2.412292945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:27.503112408 +0000 UTC m=+45.694640605" watchObservedRunningTime="2026-09-10 14:20:27.504448865 +0000 UTC m=+45.695977062"
Sep 10 14:20:28.851554 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:28.851512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:28.851554 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:28.851568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:28.852110 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:28.851648    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:20:28.852110 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:28.851712    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:20:28.852110 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:28.851725    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm podName:2b0aadca-c96f-4125-addf-c701034d2576 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:36.851704953 +0000 UTC m=+55.043233130 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wcprc" (UID: "2b0aadca-c96f-4125-addf-c701034d2576") : configmap "kube-rbac-proxy" not found
Sep 10 14:20:28.852110 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:28.851764    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls podName:2b0aadca-c96f-4125-addf-c701034d2576 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:36.851748447 +0000 UTC m=+55.043276626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcprc" (UID: "2b0aadca-c96f-4125-addf-c701034d2576") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:20:31.476394 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:31.476357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:31.476394 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:31.476394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:31.476806 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:31.476517    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:20:31.476806 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:31.476526    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:20:31.476806 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:31.476602    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls podName:e8bdbb9e-6d22-4d3e-af97-c34159145a29 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:47.476580686 +0000 UTC m=+65.668108864 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls") pod "dns-default-ss76l" (UID: "e8bdbb9e-6d22-4d3e-af97-c34159145a29") : secret "dns-default-metrics-tls" not found
Sep 10 14:20:31.476806 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:31.476531    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546fb9cb8f-xckpj: secret "image-registry-tls" not found
Sep 10 14:20:31.476806 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:31.476694    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls podName:f94d471e-af4c-46f6-b086-1ffd11503f9b nodeName:}" failed. No retries permitted until 2026-09-10 14:20:47.476674704 +0000 UTC m=+65.668202883 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls") pod "image-registry-546fb9cb8f-xckpj" (UID: "f94d471e-af4c-46f6-b086-1ffd11503f9b") : secret "image-registry-tls" not found
Sep 10 14:20:36.917107 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:36.917069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:36.917107 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:36.917112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:36.917850 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:36.917251    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:20:36.917850 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:20:36.917317    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls podName:2b0aadca-c96f-4125-addf-c701034d2576 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:52.917301088 +0000 UTC m=+71.108829268 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcprc" (UID: "2b0aadca-c96f-4125-addf-c701034d2576") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:20:36.917850 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:36.917734    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b0aadca-c96f-4125-addf-c701034d2576-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:41.444276 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:41.444249    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rjjh6"
Sep 10 14:20:47.500341 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:47.500291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:47.500341 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:47.500341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:47.502590 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:47.502568    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8bdbb9e-6d22-4d3e-af97-c34159145a29-metrics-tls\") pod \"dns-default-ss76l\" (UID: \"e8bdbb9e-6d22-4d3e-af97-c34159145a29\") " pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:47.502700 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:47.502683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls\") pod \"image-registry-546fb9cb8f-xckpj\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") " pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:47.780647 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:47.780565    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bskzl\""
Sep 10 14:20:47.785697 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:47.785680    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cgxdb\""
Sep 10 14:20:47.788589 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:47.788575    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:47.794598 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:47.794580    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:48.004439 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.004394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs\") pod \"network-metrics-daemon-qvxlq\" (UID: \"50de453c-4d08-4f49-9e7e-0d5e2221e72e\") " pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:20:48.004439 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.004440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:20:48.007517 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.007492    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 14:20:48.007643 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.007600    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 14:20:48.017775 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.017753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b046f46-d48b-4025-b9f9-37c5eea82ff9-original-pull-secret\") pod \"global-pull-secret-syncer-hgkhl\" (UID: \"0b046f46-d48b-4025-b9f9-37c5eea82ff9\") " pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:20:48.017880 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.017777    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50de453c-4d08-4f49-9e7e-0d5e2221e72e-metrics-certs\") pod \"network-metrics-daemon-qvxlq\" (UID: \"50de453c-4d08-4f49-9e7e-0d5e2221e72e\") " pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:20:48.060429 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.060346    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ss76l"]
Sep 10 14:20:48.063397 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:20:48.063369    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bdbb9e_6d22_4d3e_af97_c34159145a29.slice/crio-26aa0615b4859faa91d23a40b41c0198bbc0dd21e550f05bd03616664b178072 WatchSource:0}: Error finding container 26aa0615b4859faa91d23a40b41c0198bbc0dd21e550f05bd03616664b178072: Status 404 returned error can't find the container with id 26aa0615b4859faa91d23a40b41c0198bbc0dd21e550f05bd03616664b178072
Sep 10 14:20:48.095600 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.095580    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-546fb9cb8f-xckpj"]
Sep 10 14:20:48.097574 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:20:48.097539    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94d471e_af4c_46f6_b086_1ffd11503f9b.slice/crio-5907e1e0b8236a4bb47c75f2e71cae4a634cb88990681f05435e79e1efadd142 WatchSource:0}: Error finding container 5907e1e0b8236a4bb47c75f2e71cae4a634cb88990681f05435e79e1efadd142: Status 404 returned error can't find the container with id 5907e1e0b8236a4bb47c75f2e71cae4a634cb88990681f05435e79e1efadd142
Sep 10 14:20:48.105069 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.105047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqs5\" (UniqueName: \"kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5\") pod \"network-check-target-hn9m5\" (UID: \"dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33\") " pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:48.107928 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.107909    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 14:20:48.118683 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.118667    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 14:20:48.146641 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.146615    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjqs5\" (UniqueName: \"kubernetes.io/projected/dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33-kube-api-access-fjqs5\") pod \"network-check-target-hn9m5\" (UID: \"dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33\") " pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:48.216620 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.216580    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkhl"
Sep 10 14:20:48.225849 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.225831    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c5vvv\""
Sep 10 14:20:48.232254 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.232233    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q47dh\""
Sep 10 14:20:48.234251 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.234227    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxlq"
Sep 10 14:20:48.240048 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.240028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:48.457311 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.457283    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"]
Sep 10 14:20:48.459491 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.459476    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"
Sep 10 14:20:48.462244 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.462216    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 14:20:48.463455 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.463438    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 14:20:48.463568 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.463440    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 14:20:48.463568 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.463440    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 14:20:48.469140 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.469120    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fb649cbd-tfqcv"]
Sep 10 14:20:48.470958 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.470939    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fb649cbd-tfqcv"
Sep 10 14:20:48.473332 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.473312    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 14:20:48.473332 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.473330    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2btml\""
Sep 10 14:20:48.481931 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.481906    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"]
Sep 10 14:20:48.491952 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.488789    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fb649cbd-tfqcv"]
Sep 10 14:20:48.492528 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.492507    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-546fb9cb8f-xckpj"]
Sep 10 14:20:48.497842 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.497814    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hgkhl"]
Sep 10 14:20:48.501410 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:20:48.501379    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b046f46_d48b_4025_b9f9_37c5eea82ff9.slice/crio-34750fa6786ef155b6a14e98062baab051f23c4231d4e7b436735dad8435dd00 WatchSource:0}: Error finding container 34750fa6786ef155b6a14e98062baab051f23c4231d4e7b436735dad8435dd00: Status 404 returned error can't find the container with id 34750fa6786ef155b6a14e98062baab051f23c4231d4e7b436735dad8435dd00
Sep 10 14:20:48.507506 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.507478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4gsv\" (UniqueName: \"kubernetes.io/projected/ef5355ed-2971-46e2-9f84-534e8f7fa3d6-kube-api-access-v4gsv\") pod \"klusterlet-addon-workmgr-fbfc97b-wgslh\" (UID: \"ef5355ed-2971-46e2-9f84-534e8f7fa3d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"
Sep 10 14:20:48.507610 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.507515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n6cf\" (UniqueName: \"kubernetes.io/projected/4b6fb2aa-3138-47d2-ba4e-5787793551e9-kube-api-access-7n6cf\") pod \"managed-serviceaccount-addon-agent-fb649cbd-tfqcv\" (UID: \"4b6fb2aa-3138-47d2-ba4e-5787793551e9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fb649cbd-tfqcv"
Sep 10 14:20:48.507610 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.507538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b6fb2aa-3138-47d2-ba4e-5787793551e9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-fb649cbd-tfqcv\" (UID: \"4b6fb2aa-3138-47d2-ba4e-5787793551e9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fb649cbd-tfqcv"
Sep 10 14:20:48.507718 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.507645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef5355ed-2971-46e2-9f84-534e8f7fa3d6-tmp\") pod \"klusterlet-addon-workmgr-fbfc97b-wgslh\" (UID: \"ef5355ed-2971-46e2-9f84-534e8f7fa3d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"
Sep 10 14:20:48.507718 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.507669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ef5355ed-2971-46e2-9f84-534e8f7fa3d6-klusterlet-config\") pod \"klusterlet-addon-workmgr-fbfc97b-wgslh\" (UID: \"ef5355ed-2971-46e2-9f84-534e8f7fa3d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"
Sep 10 14:20:48.524222 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.524198    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvxlq"]
Sep 10 14:20:48.525773 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:20:48.525744    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50de453c_4d08_4f49_9e7e_0d5e2221e72e.slice/crio-d0721659cff435e6172fd234369d7aafa36ab281a517621441a0157f496afa74 WatchSource:0}: Error finding container d0721659cff435e6172fd234369d7aafa36ab281a517621441a0157f496afa74: Status 404 returned error can't find the container with id d0721659cff435e6172fd234369d7aafa36ab281a517621441a0157f496afa74
Sep 10 14:20:48.531713 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.531693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hn9m5"]
Sep 10 14:20:48.535585 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:20:48.535555    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc959a31_daf7_4ef5_8e3b_8e4d5d2eec33.slice/crio-d955d7843078f7fac9c2612a5c830acc9c79e9bb19761926928f9b30b2701dcc WatchSource:0}: Error finding container d955d7843078f7fac9c2612a5c830acc9c79e9bb19761926928f9b30b2701dcc: Status 404 returned error can't find the container with id d955d7843078f7fac9c2612a5c830acc9c79e9bb19761926928f9b30b2701dcc
Sep 10 14:20:48.549526 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.549490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hgkhl" event={"ID":"0b046f46-d48b-4025-b9f9-37c5eea82ff9","Type":"ContainerStarted","Data":"34750fa6786ef155b6a14e98062baab051f23c4231d4e7b436735dad8435dd00"}
Sep 10 14:20:48.550930 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.550897    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hn9m5" event={"ID":"dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33","Type":"ContainerStarted","Data":"d955d7843078f7fac9c2612a5c830acc9c79e9bb19761926928f9b30b2701dcc"}
Sep 10 14:20:48.553154 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.553126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvxlq" event={"ID":"50de453c-4d08-4f49-9e7e-0d5e2221e72e","Type":"ContainerStarted","Data":"d0721659cff435e6172fd234369d7aafa36ab281a517621441a0157f496afa74"}
Sep 10 14:20:48.555465 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.555216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj" event={"ID":"f94d471e-af4c-46f6-b086-1ffd11503f9b","Type":"ContainerStarted","Data":"4c809310e995d9a8a742012ed3d8c4b133e906eb557fa75b8b1658eb8a4ba324"}
Sep 10 14:20:48.555465 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.555262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj" event={"ID":"f94d471e-af4c-46f6-b086-1ffd11503f9b","Type":"ContainerStarted","Data":"5907e1e0b8236a4bb47c75f2e71cae4a634cb88990681f05435e79e1efadd142"}
Sep 10 14:20:48.555465 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.555352    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:20:48.556753 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.556710    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ss76l" event={"ID":"e8bdbb9e-6d22-4d3e-af97-c34159145a29","Type":"ContainerStarted","Data":"26aa0615b4859faa91d23a40b41c0198bbc0dd21e550f05bd03616664b178072"}
Sep 10 14:20:48.588668 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.588577    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj" podStartSLOduration=35.588556485 podStartE2EDuration="35.588556485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:48.587375863 +0000 UTC m=+66.778904061" watchObservedRunningTime="2026-09-10 14:20:48.588556485 +0000 UTC m=+66.780084685"
Sep 10 14:20:48.608873 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.608841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n6cf\" (UniqueName: \"kubernetes.io/projected/4b6fb2aa-3138-47d2-ba4e-5787793551e9-kube-api-access-7n6cf\") pod \"managed-serviceaccount-addon-agent-fb649cbd-tfqcv\" (UID: \"4b6fb2aa-3138-47d2-ba4e-5787793551e9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fb649cbd-tfqcv"
Sep 10 14:20:48.609055 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.608878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b6fb2aa-3138-47d2-ba4e-5787793551e9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-fb649cbd-tfqcv\" (UID: \"4b6fb2aa-3138-47d2-ba4e-5787793551e9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fb649cbd-tfqcv"
Sep 10 14:20:48.609055 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.609002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef5355ed-2971-46e2-9f84-534e8f7fa3d6-tmp\") pod \"klusterlet-addon-workmgr-fbfc97b-wgslh\" (UID: \"ef5355ed-2971-46e2-9f84-534e8f7fa3d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"
Sep 10 14:20:48.609055 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.609027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ef5355ed-2971-46e2-9f84-534e8f7fa3d6-klusterlet-config\") pod \"klusterlet-addon-workmgr-fbfc97b-wgslh\" (UID: \"ef5355ed-2971-46e2-9f84-534e8f7fa3d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"
Sep 10 14:20:48.609232 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.609125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4gsv\" (UniqueName: \"kubernetes.io/projected/ef5355ed-2971-46e2-9f84-534e8f7fa3d6-kube-api-access-v4gsv\") pod \"klusterlet-addon-workmgr-fbfc97b-wgslh\" (UID: \"ef5355ed-2971-46e2-9f84-534e8f7fa3d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"
Sep 10 14:20:48.609892 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.609859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef5355ed-2971-46e2-9f84-534e8f7fa3d6-tmp\") pod \"klusterlet-addon-workmgr-fbfc97b-wgslh\" (UID: \"ef5355ed-2971-46e2-9f84-534e8f7fa3d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"
Sep 10 14:20:48.612357 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.612332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ef5355ed-2971-46e2-9f84-534e8f7fa3d6-klusterlet-config\") pod \"klusterlet-addon-workmgr-fbfc97b-wgslh\" (UID: \"ef5355ed-2971-46e2-9f84-534e8f7fa3d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"
Sep 10 14:20:48.612473 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.612453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b6fb2aa-3138-47d2-ba4e-5787793551e9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-fb649cbd-tfqcv\" (UID: \"4b6fb2aa-3138-47d2-ba4e-5787793551e9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fb649cbd-tfqcv"
Sep 10 14:20:48.619837 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.619794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4gsv\" (UniqueName: \"kubernetes.io/projected/ef5355ed-2971-46e2-9f84-534e8f7fa3d6-kube-api-access-v4gsv\") pod \"klusterlet-addon-workmgr-fbfc97b-wgslh\" (UID: \"ef5355ed-2971-46e2-9f84-534e8f7fa3d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"
Sep 10 14:20:48.621267 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.621239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n6cf\" (UniqueName: \"kubernetes.io/projected/4b6fb2aa-3138-47d2-ba4e-5787793551e9-kube-api-access-7n6cf\") pod \"managed-serviceaccount-addon-agent-fb649cbd-tfqcv\" (UID: \"4b6fb2aa-3138-47d2-ba4e-5787793551e9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fb649cbd-tfqcv"
Sep 10 14:20:48.768551 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.768515    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"
Sep 10 14:20:48.786472 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:48.786441    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fb649cbd-tfqcv"
Sep 10 14:20:49.048037 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:49.047936    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"]
Sep 10 14:20:49.052989 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:20:49.052804    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5355ed_2971_46e2_9f84_534e8f7fa3d6.slice/crio-bad898b4b3cf695b4c6143186e058c688737aa812c4df4b8ad4c92ee629d8720 WatchSource:0}: Error finding container bad898b4b3cf695b4c6143186e058c688737aa812c4df4b8ad4c92ee629d8720: Status 404 returned error can't find the container with id bad898b4b3cf695b4c6143186e058c688737aa812c4df4b8ad4c92ee629d8720
Sep 10 14:20:49.068099 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:49.068070    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fb649cbd-tfqcv"]
Sep 10 14:20:49.072291 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:20:49.072259    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b6fb2aa_3138_47d2_ba4e_5787793551e9.slice/crio-aef60bef4af8bed6e37288235eba14c9bee49d2afc9ae53de1e870b0b400f6ce WatchSource:0}: Error finding container aef60bef4af8bed6e37288235eba14c9bee49d2afc9ae53de1e870b0b400f6ce: Status 404 returned error can't find the container with id aef60bef4af8bed6e37288235eba14c9bee49d2afc9ae53de1e870b0b400f6ce
Sep 10 14:20:49.562043 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:49.561943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh" event={"ID":"ef5355ed-2971-46e2-9f84-534e8f7fa3d6","Type":"ContainerStarted","Data":"bad898b4b3cf695b4c6143186e058c688737aa812c4df4b8ad4c92ee629d8720"}
Sep 10 14:20:49.564472 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:49.564195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fb649cbd-tfqcv" event={"ID":"4b6fb2aa-3138-47d2-ba4e-5787793551e9","Type":"ContainerStarted","Data":"aef60bef4af8bed6e37288235eba14c9bee49d2afc9ae53de1e870b0b400f6ce"}
Sep 10 14:20:50.585990 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:50.585876    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ss76l" event={"ID":"e8bdbb9e-6d22-4d3e-af97-c34159145a29","Type":"ContainerStarted","Data":"d059ed9c3959f64702a0b8e1bf548c80fc9c8d92d2ca1c36a120e6b0b3c24ded"}
Sep 10 14:20:50.593834 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:50.593792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvxlq" event={"ID":"50de453c-4d08-4f49-9e7e-0d5e2221e72e","Type":"ContainerStarted","Data":"a496b70cff73727df074ff02fd42628c9bf052abda09f05ef4034372e56743d5"}
Sep 10 14:20:51.605806 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:51.604900    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ss76l" event={"ID":"e8bdbb9e-6d22-4d3e-af97-c34159145a29","Type":"ContainerStarted","Data":"2a82f0b4e8fdb5fee7657d89c7f70a9ca1c2515c91cca4b1e654100feea321fd"}
Sep 10 14:20:51.605806 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:51.605723    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ss76l"
Sep 10 14:20:51.609249 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:51.608991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvxlq" event={"ID":"50de453c-4d08-4f49-9e7e-0d5e2221e72e","Type":"ContainerStarted","Data":"34f2cd5332778382a0bd8b85e7a026d896c1d511f1552efef0efda08571bb24c"}
Sep 10 14:20:51.643757 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:51.643301    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ss76l" podStartSLOduration=34.341116498 podStartE2EDuration="36.643280982s" totalImagesPullingTime="2.302164484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:51.623543752 +0000 UTC m=+69.815071942" watchObservedRunningTime="2026-09-10 14:20:51.643280982 +0000 UTC m=+69.834809178"
Sep 10 14:20:51.643757 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:51.643588    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qvxlq" podStartSLOduration=67.804333989 podStartE2EDuration="1m9.643578116s" totalImagesPullingTime="1.839244127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:51.642021348 +0000 UTC m=+69.833549546" watchObservedRunningTime="2026-09-10 14:20:51.643578116 +0000 UTC m=+69.835106318"
Sep 10 14:20:52.800116 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:52.800071    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-fsffg"]
Sep 10 14:20:52.806303 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:52.806275    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg"
Sep 10 14:20:52.808897 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:52.808872    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 14:20:52.810026 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:52.810001    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 10 14:20:52.810149 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:52.810049    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 10 14:20:52.810224 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:52.810010    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mhcqq\""
Sep 10 14:20:52.810290 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:52.810265    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 14:20:52.810439 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:52.810418    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 14:20:52.814577 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:52.814351    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-fsffg"]
Sep 10 14:20:52.946478 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:52.946433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca4a60bc-3ca1-4a04-a8a5-fff546d7f370-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-fsffg\" (UID: \"ca4a60bc-3ca1-4a04-a8a5-fff546d7f370\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg"
Sep 10 14:20:52.946632 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:52.946505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctsbk\" (UniqueName: \"kubernetes.io/projected/ca4a60bc-3ca1-4a04-a8a5-fff546d7f370-kube-api-access-ctsbk\") pod \"prometheus-operator-6f4946db56-fsffg\" (UID: \"ca4a60bc-3ca1-4a04-a8a5-fff546d7f370\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg"
Sep 10 14:20:52.946632 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:52.946572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca4a60bc-3ca1-4a04-a8a5-fff546d7f370-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-fsffg\" (UID: \"ca4a60bc-3ca1-4a04-a8a5-fff546d7f370\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg"
Sep 10 14:20:52.946735 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:52.946648    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca4a60bc-3ca1-4a04-a8a5-fff546d7f370-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-fsffg\" (UID: \"ca4a60bc-3ca1-4a04-a8a5-fff546d7f370\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg"
Sep 10 14:20:52.946735 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:52.946684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:52.949226 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:52.949199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b0aadca-c96f-4125-addf-c701034d2576-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcprc\" (UID: \"2b0aadca-c96f-4125-addf-c701034d2576\") " pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:53.047446 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:53.047407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca4a60bc-3ca1-4a04-a8a5-fff546d7f370-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-fsffg\" (UID: \"ca4a60bc-3ca1-4a04-a8a5-fff546d7f370\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg"
Sep 10 14:20:53.047628 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:53.047469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca4a60bc-3ca1-4a04-a8a5-fff546d7f370-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-fsffg\" (UID: \"ca4a60bc-3ca1-4a04-a8a5-fff546d7f370\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg"
Sep 10 14:20:53.047628 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:53.047510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctsbk\" (UniqueName: \"kubernetes.io/projected/ca4a60bc-3ca1-4a04-a8a5-fff546d7f370-kube-api-access-ctsbk\") pod \"prometheus-operator-6f4946db56-fsffg\" (UID: \"ca4a60bc-3ca1-4a04-a8a5-fff546d7f370\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg"
Sep 10 14:20:53.047628 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:53.047534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca4a60bc-3ca1-4a04-a8a5-fff546d7f370-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-fsffg\" (UID: \"ca4a60bc-3ca1-4a04-a8a5-fff546d7f370\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg"
Sep 10 14:20:53.048339 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:53.048316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca4a60bc-3ca1-4a04-a8a5-fff546d7f370-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-fsffg\" (UID: \"ca4a60bc-3ca1-4a04-a8a5-fff546d7f370\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg"
Sep 10 14:20:53.050501 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:53.050433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca4a60bc-3ca1-4a04-a8a5-fff546d7f370-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-fsffg\" (UID: \"ca4a60bc-3ca1-4a04-a8a5-fff546d7f370\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg"
Sep 10 14:20:53.050616 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:53.050542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca4a60bc-3ca1-4a04-a8a5-fff546d7f370-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-fsffg\" (UID: \"ca4a60bc-3ca1-4a04-a8a5-fff546d7f370\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg"
Sep 10 14:20:53.057589 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:53.057564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctsbk\" (UniqueName: \"kubernetes.io/projected/ca4a60bc-3ca1-4a04-a8a5-fff546d7f370-kube-api-access-ctsbk\") pod \"prometheus-operator-6f4946db56-fsffg\" (UID: \"ca4a60bc-3ca1-4a04-a8a5-fff546d7f370\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg"
Sep 10 14:20:53.118370 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:53.118323    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg"
Sep 10 14:20:53.167983 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:53.167950    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mdxkp\""
Sep 10 14:20:53.176231 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:53.176204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wcprc"
Sep 10 14:20:57.057845 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.057755    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-fsffg"]
Sep 10 14:20:57.060775 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:20:57.060735    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4a60bc_3ca1_4a04_a8a5_fff546d7f370.slice/crio-80192b788888704b588a9fc1c6ec1422d5ad3381f8f27a14b9816c996ed77f10 WatchSource:0}: Error finding container 80192b788888704b588a9fc1c6ec1422d5ad3381f8f27a14b9816c996ed77f10: Status 404 returned error can't find the container with id 80192b788888704b588a9fc1c6ec1422d5ad3381f8f27a14b9816c996ed77f10
Sep 10 14:20:57.061229 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.061206    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wcprc"]
Sep 10 14:20:57.064949 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:20:57.064924    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0aadca_c96f_4125_addf_c701034d2576.slice/crio-6a0e0c0c81ab952b34b7acfd984f18db35a8451f1efa87bbd33b90ac368555f2 WatchSource:0}: Error finding container 6a0e0c0c81ab952b34b7acfd984f18db35a8451f1efa87bbd33b90ac368555f2: Status 404 returned error can't find the container with id 6a0e0c0c81ab952b34b7acfd984f18db35a8451f1efa87bbd33b90ac368555f2
Sep 10 14:20:57.631967 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.631925    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wcprc" event={"ID":"2b0aadca-c96f-4125-addf-c701034d2576","Type":"ContainerStarted","Data":"a071b99cf427e92e9d83ba278734260a080644ca02a4ea515860c83f1c1912bc"}
Sep 10 14:20:57.632170 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.632058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wcprc" event={"ID":"2b0aadca-c96f-4125-addf-c701034d2576","Type":"ContainerStarted","Data":"6a0e0c0c81ab952b34b7acfd984f18db35a8451f1efa87bbd33b90ac368555f2"}
Sep 10 14:20:57.633170 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.633142    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg" event={"ID":"ca4a60bc-3ca1-4a04-a8a5-fff546d7f370","Type":"ContainerStarted","Data":"80192b788888704b588a9fc1c6ec1422d5ad3381f8f27a14b9816c996ed77f10"}
Sep 10 14:20:57.634587 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.634568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hgkhl" event={"ID":"0b046f46-d48b-4025-b9f9-37c5eea82ff9","Type":"ContainerStarted","Data":"86fd686776cb83be7d3e650c0bdbe500623a277d17101f1af5ed59e14c316c65"}
Sep 10 14:20:57.635949 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.635930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fb649cbd-tfqcv" event={"ID":"4b6fb2aa-3138-47d2-ba4e-5787793551e9","Type":"ContainerStarted","Data":"95037826e0081acc22a9c4e1c5fef1f0feb59685f3993308877110edc5d2a9cd"}
Sep 10 14:20:57.637370 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.637350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh" event={"ID":"ef5355ed-2971-46e2-9f84-534e8f7fa3d6","Type":"ContainerStarted","Data":"65f17196a69365f94dc47d30bdde2d6d35661c3aca24c8fb539d5e0de97bbe45"}
Sep 10 14:20:57.637564 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.637542    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"
Sep 10 14:20:57.639154 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.639134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hn9m5" event={"ID":"dc959a31-daf7-4ef5-8e3b-8e4d5d2eec33","Type":"ContainerStarted","Data":"f7ef4398cdeb24e11dc5a43c350d9427587d5da7b56a6e382491019034a3e713"}
Sep 10 14:20:57.639273 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.639259    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:20:57.639521 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.639505    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh"
Sep 10 14:20:57.668234 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.668175    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fbfc97b-wgslh" podStartSLOduration=1.904436827 podStartE2EDuration="9.668161155s" totalImagesPullingTime="7.763724328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:57.667502215 +0000 UTC m=+75.859030413" watchObservedRunningTime="2026-09-10 14:20:57.668161155 +0000 UTC m=+75.859689352"
Sep 10 14:20:57.668372 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.668272    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hgkhl" podStartSLOduration=67.372167005 podStartE2EDuration="1m15.66826659s" totalImagesPullingTime="8.296099585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:57.651204305 +0000 UTC m=+75.842732504" watchObservedRunningTime="2026-09-10 14:20:57.66826659 +0000 UTC m=+75.859794799"
Sep 10 14:20:57.682083 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.682027    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fb649cbd-tfqcv" podStartSLOduration=1.958479028 podStartE2EDuration="9.681948528s" totalImagesPullingTime="7.7234695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:57.681816421 +0000 UTC m=+75.873344610" watchObservedRunningTime="2026-09-10 14:20:57.681948528 +0000 UTC m=+75.873476727"
Sep 10 14:20:57.697147 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:57.697101    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hn9m5" podStartSLOduration=67.462264315 podStartE2EDuration="1m15.697088347s" totalImagesPullingTime="8.234824032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:57.696786118 +0000 UTC m=+75.888314341" watchObservedRunningTime="2026-09-10 14:20:57.697088347 +0000 UTC m=+75.888616544"
Sep 10 14:20:58.644817 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:58.644780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wcprc" event={"ID":"2b0aadca-c96f-4125-addf-c701034d2576","Type":"ContainerStarted","Data":"400890f13171db8cc287372095bbfd5c40dd44e3968ba77fdf50ec818b8c07af"}
Sep 10 14:20:59.651031 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:59.650994    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wcprc" event={"ID":"2b0aadca-c96f-4125-addf-c701034d2576","Type":"ContainerStarted","Data":"41c4e7e4ef42ecf78f28101e23c685e8dc6519f14cb2613b016ca54f3f728e87"}
Sep 10 14:20:59.652770 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:59.652740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg" event={"ID":"ca4a60bc-3ca1-4a04-a8a5-fff546d7f370","Type":"ContainerStarted","Data":"24216a4cc4c30f968575fe8e6fa4322e56682efd73459a65a5e69feb2f4e6180"}
Sep 10 14:20:59.652843 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:59.652792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg" event={"ID":"ca4a60bc-3ca1-4a04-a8a5-fff546d7f370","Type":"ContainerStarted","Data":"9171c01901d0d5f0f965625fcf0245465d51dbd8735dee9798954e81133f5874"}
Sep 10 14:20:59.693025 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:59.692951    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wcprc" podStartSLOduration=36.34777877 podStartE2EDuration="38.69293608s" totalImagesPullingTime="2.34515731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:20:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:59.692764848 +0000 UTC m=+77.884293060" watchObservedRunningTime="2026-09-10 14:20:59.69293608 +0000 UTC m=+77.884464278"
Sep 10 14:20:59.771694 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:20:59.771636    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-fsffg" podStartSLOduration=6.164565423 podStartE2EDuration="7.771619511s" totalImagesPullingTime="1.607054088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:59.771589821 +0000 UTC m=+77.963118018" watchObservedRunningTime="2026-09-10 14:20:59.771619511 +0000 UTC m=+77.963147755"
Sep 10 14:21:02.430323 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.429985    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"]
Sep 10 14:21:02.433200 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.433181    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"
Sep 10 14:21:02.436332 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.436307    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 14:21:02.436477 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.436377    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 14:21:02.436598 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.436584    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fwdkc\""
Sep 10 14:21:02.446736 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.446701    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"]
Sep 10 14:21:02.467818 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.467791    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hdwms"]
Sep 10 14:21:02.470081 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.470054    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.472681 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.472659    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7t8jk\""
Sep 10 14:21:02.472811 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.472763    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 14:21:02.472811 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.472788    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 14:21:02.473248 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.473234    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 14:21:02.519720 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.519681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88218fe6-c5fe-4431-b745-4fb80500b3d5-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-bkwlf\" (UID: \"88218fe6-c5fe-4431-b745-4fb80500b3d5\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"
Sep 10 14:21:02.519720 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.519728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqlw2\" (UniqueName: \"kubernetes.io/projected/88218fe6-c5fe-4431-b745-4fb80500b3d5-kube-api-access-nqlw2\") pod \"openshift-state-metrics-657fd6c54-bkwlf\" (UID: \"88218fe6-c5fe-4431-b745-4fb80500b3d5\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"
Sep 10 14:21:02.519917 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.519781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88218fe6-c5fe-4431-b745-4fb80500b3d5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-bkwlf\" (UID: \"88218fe6-c5fe-4431-b745-4fb80500b3d5\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"
Sep 10 14:21:02.519917 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.519839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88218fe6-c5fe-4431-b745-4fb80500b3d5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-bkwlf\" (UID: \"88218fe6-c5fe-4431-b745-4fb80500b3d5\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"
Sep 10 14:21:02.619101 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.619072    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ss76l"
Sep 10 14:21:02.620740 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.620719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88218fe6-c5fe-4431-b745-4fb80500b3d5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-bkwlf\" (UID: \"88218fe6-c5fe-4431-b745-4fb80500b3d5\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"
Sep 10 14:21:02.620854 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.620746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-sys\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.620854 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.620769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-metrics-client-ca\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.620931 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.620872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-root\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.620931 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.620903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-node-exporter-accelerators-collector-config\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.621038 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.620929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-node-exporter-textfile\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.621038 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.620989    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7mrm\" (UniqueName: \"kubernetes.io/projected/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-kube-api-access-m7mrm\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.621038 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.621017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-node-exporter-tls\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.621038 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.621036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88218fe6-c5fe-4431-b745-4fb80500b3d5-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-bkwlf\" (UID: \"88218fe6-c5fe-4431-b745-4fb80500b3d5\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"
Sep 10 14:21:02.621211 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.621051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-node-exporter-wtmp\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.621211 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.621080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.621211 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.621189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqlw2\" (UniqueName: \"kubernetes.io/projected/88218fe6-c5fe-4431-b745-4fb80500b3d5-kube-api-access-nqlw2\") pod \"openshift-state-metrics-657fd6c54-bkwlf\" (UID: \"88218fe6-c5fe-4431-b745-4fb80500b3d5\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"
Sep 10 14:21:02.621357 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.621211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88218fe6-c5fe-4431-b745-4fb80500b3d5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-bkwlf\" (UID: \"88218fe6-c5fe-4431-b745-4fb80500b3d5\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"
Sep 10 14:21:02.621758 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.621734    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88218fe6-c5fe-4431-b745-4fb80500b3d5-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-bkwlf\" (UID: \"88218fe6-c5fe-4431-b745-4fb80500b3d5\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"
Sep 10 14:21:02.623392 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.623372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88218fe6-c5fe-4431-b745-4fb80500b3d5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-bkwlf\" (UID: \"88218fe6-c5fe-4431-b745-4fb80500b3d5\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"
Sep 10 14:21:02.623638 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.623621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88218fe6-c5fe-4431-b745-4fb80500b3d5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-bkwlf\" (UID: \"88218fe6-c5fe-4431-b745-4fb80500b3d5\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"
Sep 10 14:21:02.629548 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.629525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqlw2\" (UniqueName: \"kubernetes.io/projected/88218fe6-c5fe-4431-b745-4fb80500b3d5-kube-api-access-nqlw2\") pod \"openshift-state-metrics-657fd6c54-bkwlf\" (UID: \"88218fe6-c5fe-4431-b745-4fb80500b3d5\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"
Sep 10 14:21:02.721833 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.721719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7mrm\" (UniqueName: \"kubernetes.io/projected/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-kube-api-access-m7mrm\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.721833 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.721757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-node-exporter-tls\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.721833 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.721782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-node-exporter-wtmp\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.721833 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.721816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.722172 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.722012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-node-exporter-wtmp\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.722172 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.722140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-sys\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.722272 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.722191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-sys\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.722272 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.722204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-metrics-client-ca\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.722272 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.722232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-root\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.722272 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.722258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-node-exporter-accelerators-collector-config\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.722458 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.722387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-node-exporter-textfile\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.722458 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.722386    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-root\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.722713 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.722688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-node-exporter-textfile\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.722830 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.722791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-metrics-client-ca\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.722897 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.722881    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-node-exporter-accelerators-collector-config\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.724760 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.724735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.724918 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.724895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-node-exporter-tls\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.731009 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.730966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7mrm\" (UniqueName: \"kubernetes.io/projected/554aeb55-0a6e-4235-9490-5b9b7d7ddf55-kube-api-access-m7mrm\") pod \"node-exporter-hdwms\" (UID: \"554aeb55-0a6e-4235-9490-5b9b7d7ddf55\") " pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:02.744802 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.744778    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"
Sep 10 14:21:02.780395 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:02.780360    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hdwms"
Sep 10 14:21:03.014934 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:03.014851    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf"]
Sep 10 14:21:03.017727 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:21:03.017694    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88218fe6_c5fe_4431_b745_4fb80500b3d5.slice/crio-8be157ac10a2ea36b60d6f963f66667d6a76d408ef0c6536a9d4365e65c6cd8c WatchSource:0}: Error finding container 8be157ac10a2ea36b60d6f963f66667d6a76d408ef0c6536a9d4365e65c6cd8c: Status 404 returned error can't find the container with id 8be157ac10a2ea36b60d6f963f66667d6a76d408ef0c6536a9d4365e65c6cd8c
Sep 10 14:21:03.672467 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:03.672434    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hdwms" event={"ID":"554aeb55-0a6e-4235-9490-5b9b7d7ddf55","Type":"ContainerStarted","Data":"e7f65a1df7db70d3af1c4275783abd308ea4ebe772ead724272fdf352d09ed66"}
Sep 10 14:21:03.674805 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:03.674775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf" event={"ID":"88218fe6-c5fe-4431-b745-4fb80500b3d5","Type":"ContainerStarted","Data":"d17e8544e709ae141e18dd6ef51d1ca9e151fbd001b1e2f2fa6eb720e58f5c0e"}
Sep 10 14:21:03.674925 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:03.674831    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf" event={"ID":"88218fe6-c5fe-4431-b745-4fb80500b3d5","Type":"ContainerStarted","Data":"448570e05340273eb8180fab3a3f63dfd9c50ecd85c15470a1da425526ad96db"}
Sep 10 14:21:03.674925 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:03.674859    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf" event={"ID":"88218fe6-c5fe-4431-b745-4fb80500b3d5","Type":"ContainerStarted","Data":"8be157ac10a2ea36b60d6f963f66667d6a76d408ef0c6536a9d4365e65c6cd8c"}
Sep 10 14:21:04.681412 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:04.681319    2580 generic.go:356] "Generic (PLEG): container finished" podID="554aeb55-0a6e-4235-9490-5b9b7d7ddf55" containerID="81f920835453421f5b127c6922d1c2c954758951379bf2804f702273a8c8ef9d" exitCode=0
Sep 10 14:21:04.681849 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:04.681412    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hdwms" event={"ID":"554aeb55-0a6e-4235-9490-5b9b7d7ddf55","Type":"ContainerDied","Data":"81f920835453421f5b127c6922d1c2c954758951379bf2804f702273a8c8ef9d"}
Sep 10 14:21:04.683625 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:04.683596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf" event={"ID":"88218fe6-c5fe-4431-b745-4fb80500b3d5","Type":"ContainerStarted","Data":"27c741d318d5e55feb64c5c6480c8a4c8659bbba202f16ead79d9cc8c33232c3"}
Sep 10 14:21:04.718515 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:04.718458    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bkwlf" podStartSLOduration=1.42755322 podStartE2EDuration="2.718433369s" totalImagesPullingTime="1.290880149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:21:04.717181078 +0000 UTC m=+82.908709276" watchObservedRunningTime="2026-09-10 14:21:04.718433369 +0000 UTC m=+82.909961569"
Sep 10 14:21:05.387377 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.387339    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-78967d786b-nxr8w"]
Sep 10 14:21:05.389712 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.389693    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.392048 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.392015    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 10 14:21:05.392186 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.392166    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 10 14:21:05.392231 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.392185    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 10 14:21:05.392287 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.392246    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7ljkk\""
Sep 10 14:21:05.392506 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.392480    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 10 14:21:05.392582 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.392486    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 10 14:21:05.392582 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.392493    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-974spsksh895c\""
Sep 10 14:21:05.402534 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.402515    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78967d786b-nxr8w"]
Sep 10 14:21:05.546227 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.546197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkk52\" (UniqueName: \"kubernetes.io/projected/f8ceed02-deb5-4069-abfc-b53ea2636458-kube-api-access-fkk52\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.546412 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.546245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-thanos-querier-tls\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.546412 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.546272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.546412 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.546293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.546412 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.546312    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.546412 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.546342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.546566 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.546419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8ceed02-deb5-4069-abfc-b53ea2636458-metrics-client-ca\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.546566 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.546453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-grpc-tls\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.647395 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.647295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-thanos-querier-tls\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.647395 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.647340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.647395 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.647365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.647395 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.647393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.647710 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.647432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.647710 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.647481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8ceed02-deb5-4069-abfc-b53ea2636458-metrics-client-ca\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.647710 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.647505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-grpc-tls\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.647710 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.647549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkk52\" (UniqueName: \"kubernetes.io/projected/f8ceed02-deb5-4069-abfc-b53ea2636458-kube-api-access-fkk52\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.648365 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.648332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8ceed02-deb5-4069-abfc-b53ea2636458-metrics-client-ca\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.650260 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.650236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.650350 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.650272    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-grpc-tls\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.650429 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.650410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.650463 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.650453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.650568 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.650546    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-thanos-querier-tls\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.650600 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.650570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8ceed02-deb5-4069-abfc-b53ea2636458-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.655749 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.655727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkk52\" (UniqueName: \"kubernetes.io/projected/f8ceed02-deb5-4069-abfc-b53ea2636458-kube-api-access-fkk52\") pod \"thanos-querier-78967d786b-nxr8w\" (UID: \"f8ceed02-deb5-4069-abfc-b53ea2636458\") " pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.688759 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.688728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hdwms" event={"ID":"554aeb55-0a6e-4235-9490-5b9b7d7ddf55","Type":"ContainerStarted","Data":"2d4bf6bbab87852c51db7e7e96ba5d80726d13d64bbc4ffe2158d89b0b72795b"}
Sep 10 14:21:05.689116 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.688778    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hdwms" event={"ID":"554aeb55-0a6e-4235-9490-5b9b7d7ddf55","Type":"ContainerStarted","Data":"73e763cbe63ad4a01711cb8b8174782b8233e6df72f5662413b60091acb0d9aa"}
Sep 10 14:21:05.699803 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.699773    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:05.711398 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.711347    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hdwms" podStartSLOduration=2.288695296 podStartE2EDuration="3.711330599s" totalImagesPullingTime="1.351413236s" totalInitContainerRuntime="71.222067ms" isStatefulPod=true podCreationTimestamp="2026-09-10 14:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:21:05.710678929 +0000 UTC m=+83.902207130" watchObservedRunningTime="2026-09-10 14:21:05.711330599 +0000 UTC m=+83.902858797"
Sep 10 14:21:05.975532 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:05.975506    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78967d786b-nxr8w"]
Sep 10 14:21:05.977581 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:21:05.977550    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ceed02_deb5_4069_abfc_b53ea2636458.slice/crio-f02840b46494680c5e382dbefda73eb45f51037e4d3d7805d2da3faf234126bd WatchSource:0}: Error finding container f02840b46494680c5e382dbefda73eb45f51037e4d3d7805d2da3faf234126bd: Status 404 returned error can't find the container with id f02840b46494680c5e382dbefda73eb45f51037e4d3d7805d2da3faf234126bd
Sep 10 14:21:06.693248 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:06.693208    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w" event={"ID":"f8ceed02-deb5-4069-abfc-b53ea2636458","Type":"ContainerStarted","Data":"f02840b46494680c5e382dbefda73eb45f51037e4d3d7805d2da3faf234126bd"}
Sep 10 14:21:07.006728 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.006648    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb"]
Sep 10 14:21:07.008925 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.008896    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb"
Sep 10 14:21:07.011824 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.011536    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 10 14:21:07.011824 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.011629    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ndnq9\""
Sep 10 14:21:07.019710 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.019682    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb"]
Sep 10 14:21:07.158439 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.158406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/837d6172-d2dd-43e7-90d2-ab57eb64d599-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-nvfxb\" (UID: \"837d6172-d2dd-43e7-90d2-ab57eb64d599\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb"
Sep 10 14:21:07.158602 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.158466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4c76\" (UniqueName: \"kubernetes.io/projected/837d6172-d2dd-43e7-90d2-ab57eb64d599-kube-api-access-p4c76\") pod \"monitoring-plugin-85bbbbbf4-nvfxb\" (UID: \"837d6172-d2dd-43e7-90d2-ab57eb64d599\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb"
Sep 10 14:21:07.259010 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.258905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/837d6172-d2dd-43e7-90d2-ab57eb64d599-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-nvfxb\" (UID: \"837d6172-d2dd-43e7-90d2-ab57eb64d599\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb"
Sep 10 14:21:07.259163 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.259007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4c76\" (UniqueName: \"kubernetes.io/projected/837d6172-d2dd-43e7-90d2-ab57eb64d599-kube-api-access-p4c76\") pod \"monitoring-plugin-85bbbbbf4-nvfxb\" (UID: \"837d6172-d2dd-43e7-90d2-ab57eb64d599\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb"
Sep 10 14:21:07.259163 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:21:07.259151    2580 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 10 14:21:07.259274 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:21:07.259220    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837d6172-d2dd-43e7-90d2-ab57eb64d599-monitoring-plugin-cert podName:837d6172-d2dd-43e7-90d2-ab57eb64d599 nodeName:}" failed. No retries permitted until 2026-09-10 14:21:07.759198729 +0000 UTC m=+85.950726905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/837d6172-d2dd-43e7-90d2-ab57eb64d599-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-nvfxb" (UID: "837d6172-d2dd-43e7-90d2-ab57eb64d599") : secret "monitoring-plugin-cert" not found
Sep 10 14:21:07.269288 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.269254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4c76\" (UniqueName: \"kubernetes.io/projected/837d6172-d2dd-43e7-90d2-ab57eb64d599-kube-api-access-p4c76\") pod \"monitoring-plugin-85bbbbbf4-nvfxb\" (UID: \"837d6172-d2dd-43e7-90d2-ab57eb64d599\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb"
Sep 10 14:21:07.590271 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.590193    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"]
Sep 10 14:21:07.592623 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.592602    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.595101 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.595071    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 10 14:21:07.595208 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.595101    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 10 14:21:07.595208 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.595170    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 10 14:21:07.595208 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.595103    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 10 14:21:07.595391 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.595359    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 10 14:21:07.595573 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.595478    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tqmfw\""
Sep 10 14:21:07.601394 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.601367    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 10 14:21:07.607404 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.607384    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"]
Sep 10 14:21:07.663257 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.663222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e8413b80-c9a4-4ce9-be64-0c02d31741d1-secret-telemeter-client\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.663399 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.663283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8413b80-c9a4-4ce9-be64-0c02d31741d1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.663470 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.663401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8413b80-c9a4-4ce9-be64-0c02d31741d1-serving-certs-ca-bundle\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.663470 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.663439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e8413b80-c9a4-4ce9-be64-0c02d31741d1-telemeter-client-tls\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.663536 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.663476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8413b80-c9a4-4ce9-be64-0c02d31741d1-metrics-client-ca\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.663591 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.663575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8413b80-c9a4-4ce9-be64-0c02d31741d1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.663638 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.663622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq9cw\" (UniqueName: \"kubernetes.io/projected/e8413b80-c9a4-4ce9-be64-0c02d31741d1-kube-api-access-qq9cw\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.663683 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.663666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e8413b80-c9a4-4ce9-be64-0c02d31741d1-federate-client-tls\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.764677 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.764640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8413b80-c9a4-4ce9-be64-0c02d31741d1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.765168 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.764700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq9cw\" (UniqueName: \"kubernetes.io/projected/e8413b80-c9a4-4ce9-be64-0c02d31741d1-kube-api-access-qq9cw\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.765168 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.764739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e8413b80-c9a4-4ce9-be64-0c02d31741d1-federate-client-tls\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.765168 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.764774    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e8413b80-c9a4-4ce9-be64-0c02d31741d1-secret-telemeter-client\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.765168 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.764809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/837d6172-d2dd-43e7-90d2-ab57eb64d599-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-nvfxb\" (UID: \"837d6172-d2dd-43e7-90d2-ab57eb64d599\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb"
Sep 10 14:21:07.765168 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.764844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8413b80-c9a4-4ce9-be64-0c02d31741d1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.765168 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.764875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8413b80-c9a4-4ce9-be64-0c02d31741d1-serving-certs-ca-bundle\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.765168 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.764905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e8413b80-c9a4-4ce9-be64-0c02d31741d1-telemeter-client-tls\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.765168 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.764928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8413b80-c9a4-4ce9-be64-0c02d31741d1-metrics-client-ca\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.765709 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.765685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8413b80-c9a4-4ce9-be64-0c02d31741d1-serving-certs-ca-bundle\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.765806 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.765685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8413b80-c9a4-4ce9-be64-0c02d31741d1-metrics-client-ca\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.765806 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.765686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8413b80-c9a4-4ce9-be64-0c02d31741d1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.767898 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.767844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e8413b80-c9a4-4ce9-be64-0c02d31741d1-federate-client-tls\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.767898 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.767842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e8413b80-c9a4-4ce9-be64-0c02d31741d1-telemeter-client-tls\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.768257 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.768231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/837d6172-d2dd-43e7-90d2-ab57eb64d599-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-nvfxb\" (UID: \"837d6172-d2dd-43e7-90d2-ab57eb64d599\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb"
Sep 10 14:21:07.768716 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.768693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8413b80-c9a4-4ce9-be64-0c02d31741d1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.768822 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.768801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e8413b80-c9a4-4ce9-be64-0c02d31741d1-secret-telemeter-client\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.773225 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.773196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq9cw\" (UniqueName: \"kubernetes.io/projected/e8413b80-c9a4-4ce9-be64-0c02d31741d1-kube-api-access-qq9cw\") pod \"telemeter-client-5bf9f95fc5-6bgxv\" (UID: \"e8413b80-c9a4-4ce9-be64-0c02d31741d1\") " pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.904621 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.904588    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"
Sep 10 14:21:07.925353 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:07.925320    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb"
Sep 10 14:21:08.399600 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:08.399570    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb"]
Sep 10 14:21:08.402795 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:21:08.402765    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837d6172_d2dd_43e7_90d2_ab57eb64d599.slice/crio-8d79b75c97ffc701e2dee5a3e741a75fdfef382a1090fb8d0015e7d0790eaba5 WatchSource:0}: Error finding container 8d79b75c97ffc701e2dee5a3e741a75fdfef382a1090fb8d0015e7d0790eaba5: Status 404 returned error can't find the container with id 8d79b75c97ffc701e2dee5a3e741a75fdfef382a1090fb8d0015e7d0790eaba5
Sep 10 14:21:08.403453 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:08.403374    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv"]
Sep 10 14:21:08.408482 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:21:08.408415    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8413b80_c9a4_4ce9_be64_0c02d31741d1.slice/crio-b39a96084f11f6cbc7a4693bab97793818677b1c97a7cf8005982b4f05729023 WatchSource:0}: Error finding container b39a96084f11f6cbc7a4693bab97793818677b1c97a7cf8005982b4f05729023: Status 404 returned error can't find the container with id b39a96084f11f6cbc7a4693bab97793818677b1c97a7cf8005982b4f05729023
Sep 10 14:21:08.706467 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:08.706349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w" event={"ID":"f8ceed02-deb5-4069-abfc-b53ea2636458","Type":"ContainerStarted","Data":"bc9b9e98f9ab6a15ac6b19f1866e0ca2b7c9f12238d90863fce4f84d7eecf2f2"}
Sep 10 14:21:08.706467 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:08.706465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w" event={"ID":"f8ceed02-deb5-4069-abfc-b53ea2636458","Type":"ContainerStarted","Data":"d6aaecf0ac938b7ece18d37a2e8551c2de3ff431a27aa5c42708e4e97ad29c78"}
Sep 10 14:21:08.706694 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:08.706503    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w" event={"ID":"f8ceed02-deb5-4069-abfc-b53ea2636458","Type":"ContainerStarted","Data":"9a3679a512a016704cf546ec3db9e25145bc3e735007a877de80f986456e8c65"}
Sep 10 14:21:08.707717 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:08.707688    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv" event={"ID":"e8413b80-c9a4-4ce9-be64-0c02d31741d1","Type":"ContainerStarted","Data":"b39a96084f11f6cbc7a4693bab97793818677b1c97a7cf8005982b4f05729023"}
Sep 10 14:21:08.708880 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:08.708860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb" event={"ID":"837d6172-d2dd-43e7-90d2-ab57eb64d599","Type":"ContainerStarted","Data":"8d79b75c97ffc701e2dee5a3e741a75fdfef382a1090fb8d0015e7d0790eaba5"}
Sep 10 14:21:09.573305 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:09.573227    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:21:09.716843 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:09.716801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w" event={"ID":"f8ceed02-deb5-4069-abfc-b53ea2636458","Type":"ContainerStarted","Data":"43216d10a365b243d5ab9ecd845402c7b9ed171db58a8d5275377cbe7507464d"}
Sep 10 14:21:09.717022 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:09.716886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w" event={"ID":"f8ceed02-deb5-4069-abfc-b53ea2636458","Type":"ContainerStarted","Data":"83980f49bd7f3f9df79fb3c5c7ef55a6e6ba0584625e395f56da25855626c0c0"}
Sep 10 14:21:09.717022 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:09.716924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w" event={"ID":"f8ceed02-deb5-4069-abfc-b53ea2636458","Type":"ContainerStarted","Data":"33bee121110407ede31c79cf91d865c7a84d889de055269454dd4eafc40b75ba"}
Sep 10 14:21:09.717022 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:09.716965    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:09.741577 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:09.741521    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w" podStartSLOduration=1.581341037 podStartE2EDuration="4.741499301s" totalImagesPullingTime="3.160158264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:21:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:21:09.73949712 +0000 UTC m=+87.931025517" watchObservedRunningTime="2026-09-10 14:21:09.741499301 +0000 UTC m=+87.933027498"
Sep 10 14:21:10.723132 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:10.723093    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv" event={"ID":"e8413b80-c9a4-4ce9-be64-0c02d31741d1","Type":"ContainerStarted","Data":"30d5713e95faf8abfb89e891a65466f959f8941e71f6e9f1cf3f281732c55eac"}
Sep 10 14:21:10.724695 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:10.724667    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb" event={"ID":"837d6172-d2dd-43e7-90d2-ab57eb64d599","Type":"ContainerStarted","Data":"2c2b7405aeba110bfd669c5a3de919d55d09e7e63e63955106915d9430146fa5"}
Sep 10 14:21:10.725159 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:10.725143    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb"
Sep 10 14:21:10.729561 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:10.729539    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb"
Sep 10 14:21:10.745868 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:10.745826    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nvfxb" podStartSLOduration=2.5770970330000003 podStartE2EDuration="4.745812164s" totalImagesPullingTime="2.168715131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:21:10.744488844 +0000 UTC m=+88.936017041" watchObservedRunningTime="2026-09-10 14:21:10.745812164 +0000 UTC m=+88.937340362"
Sep 10 14:21:11.730687 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:11.730657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv" event={"ID":"e8413b80-c9a4-4ce9-be64-0c02d31741d1","Type":"ContainerStarted","Data":"9a37f23fa8dbd04fcd2f9687502ee817d05d8e9e04dea6cf5a5c567af6e3fd04"}
Sep 10 14:21:12.736566 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:12.736534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv" event={"ID":"e8413b80-c9a4-4ce9-be64-0c02d31741d1","Type":"ContainerStarted","Data":"cffc535f246ca1984641b20eb66b7be3413a893ca51b56d6953f9382feb32a85"}
Sep 10 14:21:12.769055 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:12.768985    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5bf9f95fc5-6bgxv" podStartSLOduration=2.610502683 podStartE2EDuration="5.768949777s" totalImagesPullingTime="3.158447094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:21:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:21:12.767953577 +0000 UTC m=+90.959481777" watchObservedRunningTime="2026-09-10 14:21:12.768949777 +0000 UTC m=+90.960477976"
Sep 10 14:21:14.594734 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.594662    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj" podUID="f94d471e-af4c-46f6-b086-1ffd11503f9b" containerName="registry" containerID="cri-o://4c809310e995d9a8a742012ed3d8c4b133e906eb557fa75b8b1658eb8a4ba324" gracePeriod=30
Sep 10 14:21:14.746289 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.746258    2580 generic.go:356] "Generic (PLEG): container finished" podID="f94d471e-af4c-46f6-b086-1ffd11503f9b" containerID="4c809310e995d9a8a742012ed3d8c4b133e906eb557fa75b8b1658eb8a4ba324" exitCode=0
Sep 10 14:21:14.746423 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.746334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj" event={"ID":"f94d471e-af4c-46f6-b086-1ffd11503f9b","Type":"ContainerDied","Data":"4c809310e995d9a8a742012ed3d8c4b133e906eb557fa75b8b1658eb8a4ba324"}
Sep 10 14:21:14.830443 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.830418    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:21:14.928861 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.928834    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhqxl\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-kube-api-access-mhqxl\") pod \"f94d471e-af4c-46f6-b086-1ffd11503f9b\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") "
Sep 10 14:21:14.929057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.928868    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f94d471e-af4c-46f6-b086-1ffd11503f9b-trusted-ca\") pod \"f94d471e-af4c-46f6-b086-1ffd11503f9b\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") "
Sep 10 14:21:14.929057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.928901    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f94d471e-af4c-46f6-b086-1ffd11503f9b-ca-trust-extracted\") pod \"f94d471e-af4c-46f6-b086-1ffd11503f9b\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") "
Sep 10 14:21:14.929057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.928930    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-bound-sa-token\") pod \"f94d471e-af4c-46f6-b086-1ffd11503f9b\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") "
Sep 10 14:21:14.929057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.928958    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls\") pod \"f94d471e-af4c-46f6-b086-1ffd11503f9b\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") "
Sep 10 14:21:14.929057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.928998    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f94d471e-af4c-46f6-b086-1ffd11503f9b-installation-pull-secrets\") pod \"f94d471e-af4c-46f6-b086-1ffd11503f9b\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") "
Sep 10 14:21:14.929057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.929054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f94d471e-af4c-46f6-b086-1ffd11503f9b-image-registry-private-configuration\") pod \"f94d471e-af4c-46f6-b086-1ffd11503f9b\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") "
Sep 10 14:21:14.929360 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.929084    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-certificates\") pod \"f94d471e-af4c-46f6-b086-1ffd11503f9b\" (UID: \"f94d471e-af4c-46f6-b086-1ffd11503f9b\") "
Sep 10 14:21:14.929439 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.929405    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94d471e-af4c-46f6-b086-1ffd11503f9b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f94d471e-af4c-46f6-b086-1ffd11503f9b" (UID: "f94d471e-af4c-46f6-b086-1ffd11503f9b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:21:14.929662 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.929637    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f94d471e-af4c-46f6-b086-1ffd11503f9b" (UID: "f94d471e-af4c-46f6-b086-1ffd11503f9b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:21:14.931578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.931541    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f94d471e-af4c-46f6-b086-1ffd11503f9b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f94d471e-af4c-46f6-b086-1ffd11503f9b" (UID: "f94d471e-af4c-46f6-b086-1ffd11503f9b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:21:14.931700 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.931670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f94d471e-af4c-46f6-b086-1ffd11503f9b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f94d471e-af4c-46f6-b086-1ffd11503f9b" (UID: "f94d471e-af4c-46f6-b086-1ffd11503f9b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:21:14.931759 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.931734    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f94d471e-af4c-46f6-b086-1ffd11503f9b" (UID: "f94d471e-af4c-46f6-b086-1ffd11503f9b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:21:14.931842 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.931820    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-kube-api-access-mhqxl" (OuterVolumeSpecName: "kube-api-access-mhqxl") pod "f94d471e-af4c-46f6-b086-1ffd11503f9b" (UID: "f94d471e-af4c-46f6-b086-1ffd11503f9b"). InnerVolumeSpecName "kube-api-access-mhqxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:21:14.931920 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.931905    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f94d471e-af4c-46f6-b086-1ffd11503f9b" (UID: "f94d471e-af4c-46f6-b086-1ffd11503f9b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:21:14.940106 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:14.940075    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94d471e-af4c-46f6-b086-1ffd11503f9b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f94d471e-af4c-46f6-b086-1ffd11503f9b" (UID: "f94d471e-af4c-46f6-b086-1ffd11503f9b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:21:15.030489 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:15.030449    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhqxl\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-kube-api-access-mhqxl\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:21:15.030489 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:15.030481    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f94d471e-af4c-46f6-b086-1ffd11503f9b-trusted-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:21:15.030489 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:15.030492    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f94d471e-af4c-46f6-b086-1ffd11503f9b-ca-trust-extracted\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:21:15.030489 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:15.030500    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-bound-sa-token\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:21:15.030748 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:15.030508    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:21:15.030748 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:15.030515    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f94d471e-af4c-46f6-b086-1ffd11503f9b-installation-pull-secrets\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:21:15.030748 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:15.030524    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f94d471e-af4c-46f6-b086-1ffd11503f9b-image-registry-private-configuration\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:21:15.030748 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:15.030533    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f94d471e-af4c-46f6-b086-1ffd11503f9b-registry-certificates\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:21:15.732241 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:15.732212    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-78967d786b-nxr8w"
Sep 10 14:21:15.751381 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:15.751346    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj" event={"ID":"f94d471e-af4c-46f6-b086-1ffd11503f9b","Type":"ContainerDied","Data":"5907e1e0b8236a4bb47c75f2e71cae4a634cb88990681f05435e79e1efadd142"}
Sep 10 14:21:15.751528 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:15.751408    2580 scope.go:117] "RemoveContainer" containerID="4c809310e995d9a8a742012ed3d8c4b133e906eb557fa75b8b1658eb8a4ba324"
Sep 10 14:21:15.751528 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:15.751357    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546fb9cb8f-xckpj"
Sep 10 14:21:15.774658 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:15.774632    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-546fb9cb8f-xckpj"]
Sep 10 14:21:15.778382 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:15.778359    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-546fb9cb8f-xckpj"]
Sep 10 14:21:16.308390 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:16.308360    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f94d471e-af4c-46f6-b086-1ffd11503f9b" path="/var/lib/kubelet/pods/f94d471e-af4c-46f6-b086-1ffd11503f9b/volumes"
Sep 10 14:21:22.205917 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.205884    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68f6f778b4-fxmtm"]
Sep 10 14:21:22.206286 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.206182    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f94d471e-af4c-46f6-b086-1ffd11503f9b" containerName="registry"
Sep 10 14:21:22.206286 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.206193    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f94d471e-af4c-46f6-b086-1ffd11503f9b" containerName="registry"
Sep 10 14:21:22.206286 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.206244    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f94d471e-af4c-46f6-b086-1ffd11503f9b" containerName="registry"
Sep 10 14:21:22.209720 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.209701    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.212683 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.212659    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 14:21:22.213557 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.213541    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qcks9\""
Sep 10 14:21:22.213916 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.213891    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 14:21:22.214147 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.214125    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 14:21:22.214213 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.214147    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 14:21:22.214213 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.214138    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 14:21:22.214213 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.214166    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 14:21:22.214833 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.214817    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 14:21:22.217904 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.217884    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 14:21:22.224255 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.224236    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f6f778b4-fxmtm"]
Sep 10 14:21:22.380493 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.380446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-service-ca\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.380493 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.380494    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9ee8c86-5475-4bea-a058-2466926bc971-console-serving-cert\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.380728 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.380539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-oauth-serving-cert\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.380728 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.380598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-trusted-ca-bundle\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.380728 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.380646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9ee8c86-5475-4bea-a058-2466926bc971-console-oauth-config\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.380728 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.380666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-console-config\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.380728 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.380691    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8r8q\" (UniqueName: \"kubernetes.io/projected/a9ee8c86-5475-4bea-a058-2466926bc971-kube-api-access-k8r8q\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.481964 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.481873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9ee8c86-5475-4bea-a058-2466926bc971-console-oauth-config\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.481964 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.481913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-console-config\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.481964 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.481947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8r8q\" (UniqueName: \"kubernetes.io/projected/a9ee8c86-5475-4bea-a058-2466926bc971-kube-api-access-k8r8q\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.482226 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.481987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-service-ca\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.482226 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.482004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9ee8c86-5475-4bea-a058-2466926bc971-console-serving-cert\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.482226 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.482172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-oauth-serving-cert\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.482362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.482249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-trusted-ca-bundle\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.482813 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.482790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-oauth-serving-cert\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.482931 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.482792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-service-ca\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.482931 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.482792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-console-config\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.483203 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.483179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-trusted-ca-bundle\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.484560 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.484531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9ee8c86-5475-4bea-a058-2466926bc971-console-serving-cert\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.484656 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.484590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9ee8c86-5475-4bea-a058-2466926bc971-console-oauth-config\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.490891 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.490870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8r8q\" (UniqueName: \"kubernetes.io/projected/a9ee8c86-5475-4bea-a058-2466926bc971-kube-api-access-k8r8q\") pod \"console-68f6f778b4-fxmtm\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") " pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.519780 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.519740    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:22.791844 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:22.791770    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f6f778b4-fxmtm"]
Sep 10 14:21:22.794535 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:21:22.794506    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ee8c86_5475_4bea_a058_2466926bc971.slice/crio-352718d2a91a330b7f61088604567c4bab2caa08628976022b276bb5168a2d7b WatchSource:0}: Error finding container 352718d2a91a330b7f61088604567c4bab2caa08628976022b276bb5168a2d7b: Status 404 returned error can't find the container with id 352718d2a91a330b7f61088604567c4bab2caa08628976022b276bb5168a2d7b
Sep 10 14:21:23.788218 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:23.788174    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f6f778b4-fxmtm" event={"ID":"a9ee8c86-5475-4bea-a058-2466926bc971","Type":"ContainerStarted","Data":"352718d2a91a330b7f61088604567c4bab2caa08628976022b276bb5168a2d7b"}
Sep 10 14:21:26.801496 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:26.801460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f6f778b4-fxmtm" event={"ID":"a9ee8c86-5475-4bea-a058-2466926bc971","Type":"ContainerStarted","Data":"4e713662e2b92c78765e9d55be1b59c7ba4fcad840d3a7695cfeb9cd84362c56"}
Sep 10 14:21:26.821479 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:26.821432    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68f6f778b4-fxmtm" podStartSLOduration=1.259011057 podStartE2EDuration="4.821417485s" totalImagesPullingTime="3.562406428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:21:26.81911779 +0000 UTC m=+105.010645990" watchObservedRunningTime="2026-09-10 14:21:26.821417485 +0000 UTC m=+105.012945683"
Sep 10 14:21:28.647619 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:28.647584    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hn9m5"
Sep 10 14:21:32.519999 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:32.519951    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:32.520369 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:32.520021    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:32.524862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:32.524837    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:32.828107 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:32.828016    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:21:41.133206 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:21:41.133183    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6z7bn_bd2cae1f-0990-4824-857c-a1872793416d/serve-healthcheck-canary/0.log"
Sep 10 14:22:24.904989 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:24.904928    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 14:22:24.908020 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:24.907998    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:24.910466 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:24.910431    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 14:22:24.910579 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:24.910499    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 14:22:24.910579 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:24.910509    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 14:22:24.910727 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:24.910713    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 14:22:24.910802 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:24.910780    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 14:22:24.910921 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:24.910853    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 14:22:24.910921 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:24.910907    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 14:22:24.911059 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:24.910992    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 14:22:24.911059 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:24.911010    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t9cwl\""
Sep 10 14:22:24.915224 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:24.915206    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 14:22:24.923538 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:24.923515    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 14:22:25.076607 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.076570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.076607 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.076606    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.076832 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.076632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/563ccb51-a496-4cca-b591-a5ed88e98f4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.076832 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.076689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf284\" (UniqueName: \"kubernetes.io/projected/563ccb51-a496-4cca-b591-a5ed88e98f4a-kube-api-access-mf284\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.076832 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.076783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.076832 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.076815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/563ccb51-a496-4cca-b591-a5ed88e98f4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.076832 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.076834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.077094 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.076889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.077094 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.076930    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.077094 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.076965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/563ccb51-a496-4cca-b591-a5ed88e98f4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.077094 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.077027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/563ccb51-a496-4cca-b591-a5ed88e98f4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.077094 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.077060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/563ccb51-a496-4cca-b591-a5ed88e98f4a-config-out\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.077094 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.077087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-web-config\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.177643 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.177568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.177643 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.177599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.177643 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.177619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/563ccb51-a496-4cca-b591-a5ed88e98f4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.177643 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.177639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/563ccb51-a496-4cca-b591-a5ed88e98f4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.177919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.177668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/563ccb51-a496-4cca-b591-a5ed88e98f4a-config-out\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.177919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.177701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-web-config\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.177919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.177738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.177919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.177762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.177919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.177780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/563ccb51-a496-4cca-b591-a5ed88e98f4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.177919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.177798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf284\" (UniqueName: \"kubernetes.io/projected/563ccb51-a496-4cca-b591-a5ed88e98f4a-kube-api-access-mf284\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.177919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.177847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.177919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.177874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/563ccb51-a496-4cca-b591-a5ed88e98f4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.177919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.177891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.178377 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.178073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/563ccb51-a496-4cca-b591-a5ed88e98f4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.178916 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.178892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/563ccb51-a496-4cca-b591-a5ed88e98f4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.181106 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.180932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-web-config\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.181106 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.180946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.181106 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.181044    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.181313 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.181195    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/563ccb51-a496-4cca-b591-a5ed88e98f4a-config-out\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.181313 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.181203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.181313 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.181236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.181545 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.181522    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.181615 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.181556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/563ccb51-a496-4cca-b591-a5ed88e98f4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.182031 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.182007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/563ccb51-a496-4cca-b591-a5ed88e98f4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.182903 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.182884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/563ccb51-a496-4cca-b591-a5ed88e98f4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.186013 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.185989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf284\" (UniqueName: \"kubernetes.io/projected/563ccb51-a496-4cca-b591-a5ed88e98f4a-kube-api-access-mf284\") pod \"alertmanager-main-0\" (UID: \"563ccb51-a496-4cca-b591-a5ed88e98f4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.217915 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.217874    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 14:22:25.499501 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:25.499478    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 14:22:25.500765 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:22:25.500738    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563ccb51_a496_4cca_b591_a5ed88e98f4a.slice/crio-30073717c746ba2d90cdd47fa5966977630684066fa3393747e06b9020e98ecd WatchSource:0}: Error finding container 30073717c746ba2d90cdd47fa5966977630684066fa3393747e06b9020e98ecd: Status 404 returned error can't find the container with id 30073717c746ba2d90cdd47fa5966977630684066fa3393747e06b9020e98ecd
Sep 10 14:22:26.022337 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:26.022307    2580 generic.go:356] "Generic (PLEG): container finished" podID="563ccb51-a496-4cca-b591-a5ed88e98f4a" containerID="0d1f13261f5d163b3673b191f5194bf014995cab1d05306ab68234987a82cb7f" exitCode=0
Sep 10 14:22:26.022719 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:26.022340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563ccb51-a496-4cca-b591-a5ed88e98f4a","Type":"ContainerDied","Data":"0d1f13261f5d163b3673b191f5194bf014995cab1d05306ab68234987a82cb7f"}
Sep 10 14:22:26.022719 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:26.022389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563ccb51-a496-4cca-b591-a5ed88e98f4a","Type":"ContainerStarted","Data":"30073717c746ba2d90cdd47fa5966977630684066fa3393747e06b9020e98ecd"}
Sep 10 14:22:28.034999 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:28.034932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563ccb51-a496-4cca-b591-a5ed88e98f4a","Type":"ContainerStarted","Data":"cd2636ec22b0f463b3ad8c84d4794a1709122e4ab75b299867ef3433bf4fc703"}
Sep 10 14:22:28.035384 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:28.035011    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563ccb51-a496-4cca-b591-a5ed88e98f4a","Type":"ContainerStarted","Data":"cf7e687fb57a02d72ab9aa0fac6f8618e6cc59f0ca98db4eae53c7efe8ce7bb4"}
Sep 10 14:22:28.035384 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:28.035040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563ccb51-a496-4cca-b591-a5ed88e98f4a","Type":"ContainerStarted","Data":"fd687985c5904d2b6c0138b185d3b726a59d9de354da72f3561290d09785d9ec"}
Sep 10 14:22:28.035384 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:28.035064    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563ccb51-a496-4cca-b591-a5ed88e98f4a","Type":"ContainerStarted","Data":"c97748d2ed64891dfc02c262f62a0c16fc9186b6dd1b488d3b616a5f397bed6f"}
Sep 10 14:22:28.035384 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:28.035086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563ccb51-a496-4cca-b591-a5ed88e98f4a","Type":"ContainerStarted","Data":"bd2218eff0d608b7f065b30cd52dd6b2ad4ba9c614a840cdbb1cf0c32851c30a"}
Sep 10 14:22:29.042270 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:29.042228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"563ccb51-a496-4cca-b591-a5ed88e98f4a","Type":"ContainerStarted","Data":"a711e7503f855e3ab8aa54e1f0193f482206c5af6a79ac5bd7a3ed26b26ea07f"}
Sep 10 14:22:29.075421 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:29.075369    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.341645185 podStartE2EDuration="5.075352905s" totalImagesPullingTime="1.715076081s" totalInitContainerRuntime="18.631639ms" isStatefulPod=false podCreationTimestamp="2026-09-10 14:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:22:29.07336787 +0000 UTC m=+167.264896108" watchObservedRunningTime="2026-09-10 14:22:29.075352905 +0000 UTC m=+167.266881103"
Sep 10 14:22:56.676512 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:22:56.676435    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68f6f778b4-fxmtm"]
Sep 10 14:23:18.609630 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.609592    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"]
Sep 10 14:23:18.612059 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.612041    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"
Sep 10 14:23:18.614587 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.614556    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 14:23:18.615460 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.615439    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-clm7d\""
Sep 10 14:23:18.615460 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.615454    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 14:23:18.616988 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.616952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06576458-68d8-4f9f-972f-255aecc8e5bd-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm\" (UID: \"06576458-68d8-4f9f-972f-255aecc8e5bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"
Sep 10 14:23:18.617087 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.617027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwfjv\" (UniqueName: \"kubernetes.io/projected/06576458-68d8-4f9f-972f-255aecc8e5bd-kube-api-access-fwfjv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm\" (UID: \"06576458-68d8-4f9f-972f-255aecc8e5bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"
Sep 10 14:23:18.617087 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.617053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06576458-68d8-4f9f-972f-255aecc8e5bd-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm\" (UID: \"06576458-68d8-4f9f-972f-255aecc8e5bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"
Sep 10 14:23:18.620869 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.620848    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"]
Sep 10 14:23:18.718269 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.718235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwfjv\" (UniqueName: \"kubernetes.io/projected/06576458-68d8-4f9f-972f-255aecc8e5bd-kube-api-access-fwfjv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm\" (UID: \"06576458-68d8-4f9f-972f-255aecc8e5bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"
Sep 10 14:23:18.718465 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.718276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06576458-68d8-4f9f-972f-255aecc8e5bd-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm\" (UID: \"06576458-68d8-4f9f-972f-255aecc8e5bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"
Sep 10 14:23:18.718465 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.718327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06576458-68d8-4f9f-972f-255aecc8e5bd-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm\" (UID: \"06576458-68d8-4f9f-972f-255aecc8e5bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"
Sep 10 14:23:18.718662 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.718642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06576458-68d8-4f9f-972f-255aecc8e5bd-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm\" (UID: \"06576458-68d8-4f9f-972f-255aecc8e5bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"
Sep 10 14:23:18.718715 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.718671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06576458-68d8-4f9f-972f-255aecc8e5bd-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm\" (UID: \"06576458-68d8-4f9f-972f-255aecc8e5bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"
Sep 10 14:23:18.726873 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.726839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwfjv\" (UniqueName: \"kubernetes.io/projected/06576458-68d8-4f9f-972f-255aecc8e5bd-kube-api-access-fwfjv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm\" (UID: \"06576458-68d8-4f9f-972f-255aecc8e5bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"
Sep 10 14:23:18.920782 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:18.920750    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"
Sep 10 14:23:19.189643 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:19.189568    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"]
Sep 10 14:23:19.193453 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:23:19.193425    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06576458_68d8_4f9f_972f_255aecc8e5bd.slice/crio-b5fcf507910cc2e21d1ad0ae566538832764301e958673fc0f8c3b14b3cd0d91 WatchSource:0}: Error finding container b5fcf507910cc2e21d1ad0ae566538832764301e958673fc0f8c3b14b3cd0d91: Status 404 returned error can't find the container with id b5fcf507910cc2e21d1ad0ae566538832764301e958673fc0f8c3b14b3cd0d91
Sep 10 14:23:19.234737 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:19.234706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm" event={"ID":"06576458-68d8-4f9f-972f-255aecc8e5bd","Type":"ContainerStarted","Data":"b5fcf507910cc2e21d1ad0ae566538832764301e958673fc0f8c3b14b3cd0d91"}
Sep 10 14:23:21.701110 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:21.701068    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68f6f778b4-fxmtm" podUID="a9ee8c86-5475-4bea-a058-2466926bc971" containerName="console" containerID="cri-o://4e713662e2b92c78765e9d55be1b59c7ba4fcad840d3a7695cfeb9cd84362c56" gracePeriod=15
Sep 10 14:23:21.955328 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:21.955266    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:23:22.045406 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.045374    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9ee8c86-5475-4bea-a058-2466926bc971-console-oauth-config\") pod \"a9ee8c86-5475-4bea-a058-2466926bc971\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") "
Sep 10 14:23:22.045406 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.045408    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-console-config\") pod \"a9ee8c86-5475-4bea-a058-2466926bc971\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") "
Sep 10 14:23:22.045620 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.045434    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-service-ca\") pod \"a9ee8c86-5475-4bea-a058-2466926bc971\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") "
Sep 10 14:23:22.045620 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.045450    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-oauth-serving-cert\") pod \"a9ee8c86-5475-4bea-a058-2466926bc971\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") "
Sep 10 14:23:22.045620 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.045500    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9ee8c86-5475-4bea-a058-2466926bc971-console-serving-cert\") pod \"a9ee8c86-5475-4bea-a058-2466926bc971\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") "
Sep 10 14:23:22.045620 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.045533    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-trusted-ca-bundle\") pod \"a9ee8c86-5475-4bea-a058-2466926bc971\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") "
Sep 10 14:23:22.045620 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.045572    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8r8q\" (UniqueName: \"kubernetes.io/projected/a9ee8c86-5475-4bea-a058-2466926bc971-kube-api-access-k8r8q\") pod \"a9ee8c86-5475-4bea-a058-2466926bc971\" (UID: \"a9ee8c86-5475-4bea-a058-2466926bc971\") "
Sep 10 14:23:22.045936 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.045909    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-console-config" (OuterVolumeSpecName: "console-config") pod "a9ee8c86-5475-4bea-a058-2466926bc971" (UID: "a9ee8c86-5475-4bea-a058-2466926bc971"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:23:22.046032 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.045934    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a9ee8c86-5475-4bea-a058-2466926bc971" (UID: "a9ee8c86-5475-4bea-a058-2466926bc971"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:23:22.046032 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.045944    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-service-ca" (OuterVolumeSpecName: "service-ca") pod "a9ee8c86-5475-4bea-a058-2466926bc971" (UID: "a9ee8c86-5475-4bea-a058-2466926bc971"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:23:22.046106 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.046021    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a9ee8c86-5475-4bea-a058-2466926bc971" (UID: "a9ee8c86-5475-4bea-a058-2466926bc971"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:23:22.047706 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.047675    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ee8c86-5475-4bea-a058-2466926bc971-kube-api-access-k8r8q" (OuterVolumeSpecName: "kube-api-access-k8r8q") pod "a9ee8c86-5475-4bea-a058-2466926bc971" (UID: "a9ee8c86-5475-4bea-a058-2466926bc971"). InnerVolumeSpecName "kube-api-access-k8r8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:23:22.047804 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.047719    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9ee8c86-5475-4bea-a058-2466926bc971-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a9ee8c86-5475-4bea-a058-2466926bc971" (UID: "a9ee8c86-5475-4bea-a058-2466926bc971"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:23:22.047804 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.047736    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9ee8c86-5475-4bea-a058-2466926bc971-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a9ee8c86-5475-4bea-a058-2466926bc971" (UID: "a9ee8c86-5475-4bea-a058-2466926bc971"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:23:22.146802 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.146766    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9ee8c86-5475-4bea-a058-2466926bc971-console-serving-cert\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:23:22.146802 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.146801    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-trusted-ca-bundle\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:23:22.147046 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.146822    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8r8q\" (UniqueName: \"kubernetes.io/projected/a9ee8c86-5475-4bea-a058-2466926bc971-kube-api-access-k8r8q\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:23:22.147046 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.146834    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9ee8c86-5475-4bea-a058-2466926bc971-console-oauth-config\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:23:22.147046 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.146848    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-console-config\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:23:22.147046 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.146857    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:23:22.147046 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.146864    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9ee8c86-5475-4bea-a058-2466926bc971-oauth-serving-cert\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:23:22.247480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.247390    2580 generic.go:356] "Generic (PLEG): container finished" podID="a9ee8c86-5475-4bea-a058-2466926bc971" containerID="4e713662e2b92c78765e9d55be1b59c7ba4fcad840d3a7695cfeb9cd84362c56" exitCode=0
Sep 10 14:23:22.247480 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.247448    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f6f778b4-fxmtm" event={"ID":"a9ee8c86-5475-4bea-a058-2466926bc971","Type":"ContainerDied","Data":"4e713662e2b92c78765e9d55be1b59c7ba4fcad840d3a7695cfeb9cd84362c56"}
Sep 10 14:23:22.247683 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.247488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f6f778b4-fxmtm" event={"ID":"a9ee8c86-5475-4bea-a058-2466926bc971","Type":"ContainerDied","Data":"352718d2a91a330b7f61088604567c4bab2caa08628976022b276bb5168a2d7b"}
Sep 10 14:23:22.247683 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.247453    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f6f778b4-fxmtm"
Sep 10 14:23:22.247683 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.247514    2580 scope.go:117] "RemoveContainer" containerID="4e713662e2b92c78765e9d55be1b59c7ba4fcad840d3a7695cfeb9cd84362c56"
Sep 10 14:23:22.258937 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.258912    2580 scope.go:117] "RemoveContainer" containerID="4e713662e2b92c78765e9d55be1b59c7ba4fcad840d3a7695cfeb9cd84362c56"
Sep 10 14:23:22.259464 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:23:22.259435    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e713662e2b92c78765e9d55be1b59c7ba4fcad840d3a7695cfeb9cd84362c56\": container with ID starting with 4e713662e2b92c78765e9d55be1b59c7ba4fcad840d3a7695cfeb9cd84362c56 not found: ID does not exist" containerID="4e713662e2b92c78765e9d55be1b59c7ba4fcad840d3a7695cfeb9cd84362c56"
Sep 10 14:23:22.259562 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.259473    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e713662e2b92c78765e9d55be1b59c7ba4fcad840d3a7695cfeb9cd84362c56"} err="failed to get container status \"4e713662e2b92c78765e9d55be1b59c7ba4fcad840d3a7695cfeb9cd84362c56\": rpc error: code = NotFound desc = could not find container \"4e713662e2b92c78765e9d55be1b59c7ba4fcad840d3a7695cfeb9cd84362c56\": container with ID starting with 4e713662e2b92c78765e9d55be1b59c7ba4fcad840d3a7695cfeb9cd84362c56 not found: ID does not exist"
Sep 10 14:23:22.270658 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.270628    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68f6f778b4-fxmtm"]
Sep 10 14:23:22.274204 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.274184    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68f6f778b4-fxmtm"]
Sep 10 14:23:22.310529 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:22.310497    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9ee8c86-5475-4bea-a058-2466926bc971" path="/var/lib/kubelet/pods/a9ee8c86-5475-4bea-a058-2466926bc971/volumes"
Sep 10 14:23:28.275667 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:28.275630    2580 generic.go:356] "Generic (PLEG): container finished" podID="06576458-68d8-4f9f-972f-255aecc8e5bd" containerID="507d3df9146e0a2527a68c80141d1e8ae0b2a8427b24badf2dcb3d9d0efba02d" exitCode=0
Sep 10 14:23:28.276142 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:28.275713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm" event={"ID":"06576458-68d8-4f9f-972f-255aecc8e5bd","Type":"ContainerDied","Data":"507d3df9146e0a2527a68c80141d1e8ae0b2a8427b24badf2dcb3d9d0efba02d"}
Sep 10 14:23:30.286875 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:30.286843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm" event={"ID":"06576458-68d8-4f9f-972f-255aecc8e5bd","Type":"ContainerStarted","Data":"c6550907309a39f07d46e2e27c97a58247f73c792fb82fb6f51795e03ee076cd"}
Sep 10 14:23:31.292080 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:31.292043    2580 generic.go:356] "Generic (PLEG): container finished" podID="06576458-68d8-4f9f-972f-255aecc8e5bd" containerID="c6550907309a39f07d46e2e27c97a58247f73c792fb82fb6f51795e03ee076cd" exitCode=0
Sep 10 14:23:31.292442 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:31.292137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm" event={"ID":"06576458-68d8-4f9f-972f-255aecc8e5bd","Type":"ContainerDied","Data":"c6550907309a39f07d46e2e27c97a58247f73c792fb82fb6f51795e03ee076cd"}
Sep 10 14:23:38.321941 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:38.321850    2580 generic.go:356] "Generic (PLEG): container finished" podID="06576458-68d8-4f9f-972f-255aecc8e5bd" containerID="af97c37417628052437dcd13b1f8e9dc442d4d0f0c90c09d15001eec29e63608" exitCode=0
Sep 10 14:23:38.321941 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:38.321926    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm" event={"ID":"06576458-68d8-4f9f-972f-255aecc8e5bd","Type":"ContainerDied","Data":"af97c37417628052437dcd13b1f8e9dc442d4d0f0c90c09d15001eec29e63608"}
Sep 10 14:23:39.444115 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:39.444089    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"
Sep 10 14:23:39.494626 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:39.494594    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06576458-68d8-4f9f-972f-255aecc8e5bd-util\") pod \"06576458-68d8-4f9f-972f-255aecc8e5bd\" (UID: \"06576458-68d8-4f9f-972f-255aecc8e5bd\") "
Sep 10 14:23:39.494795 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:39.494643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwfjv\" (UniqueName: \"kubernetes.io/projected/06576458-68d8-4f9f-972f-255aecc8e5bd-kube-api-access-fwfjv\") pod \"06576458-68d8-4f9f-972f-255aecc8e5bd\" (UID: \"06576458-68d8-4f9f-972f-255aecc8e5bd\") "
Sep 10 14:23:39.494795 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:39.494662    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06576458-68d8-4f9f-972f-255aecc8e5bd-bundle\") pod \"06576458-68d8-4f9f-972f-255aecc8e5bd\" (UID: \"06576458-68d8-4f9f-972f-255aecc8e5bd\") "
Sep 10 14:23:39.495337 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:39.495307    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06576458-68d8-4f9f-972f-255aecc8e5bd-bundle" (OuterVolumeSpecName: "bundle") pod "06576458-68d8-4f9f-972f-255aecc8e5bd" (UID: "06576458-68d8-4f9f-972f-255aecc8e5bd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:23:39.496834 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:39.496801    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06576458-68d8-4f9f-972f-255aecc8e5bd-kube-api-access-fwfjv" (OuterVolumeSpecName: "kube-api-access-fwfjv") pod "06576458-68d8-4f9f-972f-255aecc8e5bd" (UID: "06576458-68d8-4f9f-972f-255aecc8e5bd"). InnerVolumeSpecName "kube-api-access-fwfjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:23:39.498553 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:39.498532    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06576458-68d8-4f9f-972f-255aecc8e5bd-util" (OuterVolumeSpecName: "util") pod "06576458-68d8-4f9f-972f-255aecc8e5bd" (UID: "06576458-68d8-4f9f-972f-255aecc8e5bd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:23:39.596002 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:39.595904    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06576458-68d8-4f9f-972f-255aecc8e5bd-util\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:23:39.596002 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:39.595929    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwfjv\" (UniqueName: \"kubernetes.io/projected/06576458-68d8-4f9f-972f-255aecc8e5bd-kube-api-access-fwfjv\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:23:39.596002 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:39.595939    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06576458-68d8-4f9f-972f-255aecc8e5bd-bundle\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:23:40.332166 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:40.332141    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm"
Sep 10 14:23:40.332293 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:40.332146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n4pwm" event={"ID":"06576458-68d8-4f9f-972f-255aecc8e5bd","Type":"ContainerDied","Data":"b5fcf507910cc2e21d1ad0ae566538832764301e958673fc0f8c3b14b3cd0d91"}
Sep 10 14:23:40.332293 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:40.332248    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5fcf507910cc2e21d1ad0ae566538832764301e958673fc0f8c3b14b3cd0d91"
Sep 10 14:23:46.816607 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.816571    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hr9qp"]
Sep 10 14:23:46.816961 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.816896    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06576458-68d8-4f9f-972f-255aecc8e5bd" containerName="pull"
Sep 10 14:23:46.816961 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.816908    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="06576458-68d8-4f9f-972f-255aecc8e5bd" containerName="pull"
Sep 10 14:23:46.816961 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.816928    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06576458-68d8-4f9f-972f-255aecc8e5bd" containerName="extract"
Sep 10 14:23:46.816961 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.816933    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="06576458-68d8-4f9f-972f-255aecc8e5bd" containerName="extract"
Sep 10 14:23:46.816961 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.816942    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06576458-68d8-4f9f-972f-255aecc8e5bd" containerName="util"
Sep 10 14:23:46.816961 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.816949    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="06576458-68d8-4f9f-972f-255aecc8e5bd" containerName="util"
Sep 10 14:23:46.816961 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.816956    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ee8c86-5475-4bea-a058-2466926bc971" containerName="console"
Sep 10 14:23:46.816961 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.816961    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ee8c86-5475-4bea-a058-2466926bc971" containerName="console"
Sep 10 14:23:46.817233 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.817018    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="06576458-68d8-4f9f-972f-255aecc8e5bd" containerName="extract"
Sep 10 14:23:46.817233 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.817025    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9ee8c86-5475-4bea-a058-2466926bc971" containerName="console"
Sep 10 14:23:46.820260 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.820243    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hr9qp"
Sep 10 14:23:46.822807 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.822782    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 10 14:23:46.822932 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.822828    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-trzd9\""
Sep 10 14:23:46.822932 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.822828    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:23:46.835152 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.835132    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hr9qp"]
Sep 10 14:23:46.954278 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.954235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlmcm\" (UniqueName: \"kubernetes.io/projected/f3d99a2f-e3ee-482e-a963-62add79dbccd-kube-api-access-xlmcm\") pod \"cert-manager-operator-controller-manager-55c444bf86-hr9qp\" (UID: \"f3d99a2f-e3ee-482e-a963-62add79dbccd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hr9qp"
Sep 10 14:23:46.954452 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:46.954302    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3d99a2f-e3ee-482e-a963-62add79dbccd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hr9qp\" (UID: \"f3d99a2f-e3ee-482e-a963-62add79dbccd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hr9qp"
Sep 10 14:23:47.055169 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:47.055136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlmcm\" (UniqueName: \"kubernetes.io/projected/f3d99a2f-e3ee-482e-a963-62add79dbccd-kube-api-access-xlmcm\") pod \"cert-manager-operator-controller-manager-55c444bf86-hr9qp\" (UID: \"f3d99a2f-e3ee-482e-a963-62add79dbccd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hr9qp"
Sep 10 14:23:47.055334 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:47.055188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3d99a2f-e3ee-482e-a963-62add79dbccd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hr9qp\" (UID: \"f3d99a2f-e3ee-482e-a963-62add79dbccd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hr9qp"
Sep 10 14:23:47.055557 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:47.055539    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3d99a2f-e3ee-482e-a963-62add79dbccd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hr9qp\" (UID: \"f3d99a2f-e3ee-482e-a963-62add79dbccd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hr9qp"
Sep 10 14:23:47.064598 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:47.064567    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlmcm\" (UniqueName: \"kubernetes.io/projected/f3d99a2f-e3ee-482e-a963-62add79dbccd-kube-api-access-xlmcm\") pod \"cert-manager-operator-controller-manager-55c444bf86-hr9qp\" (UID: \"f3d99a2f-e3ee-482e-a963-62add79dbccd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hr9qp"
Sep 10 14:23:47.128937 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:47.128855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hr9qp"
Sep 10 14:23:47.426312 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:47.426273    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hr9qp"]
Sep 10 14:23:47.443219 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:23:47.443185    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d99a2f_e3ee_482e_a963_62add79dbccd.slice/crio-d16d026659e404529412350b436c8340fbd750b38b8b49f20017bf5b65a58333 WatchSource:0}: Error finding container d16d026659e404529412350b436c8340fbd750b38b8b49f20017bf5b65a58333: Status 404 returned error can't find the container with id d16d026659e404529412350b436c8340fbd750b38b8b49f20017bf5b65a58333
Sep 10 14:23:48.363674 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:48.363621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hr9qp" event={"ID":"f3d99a2f-e3ee-482e-a963-62add79dbccd","Type":"ContainerStarted","Data":"d16d026659e404529412350b436c8340fbd750b38b8b49f20017bf5b65a58333"}
Sep 10 14:23:49.369697 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:49.369603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hr9qp" event={"ID":"f3d99a2f-e3ee-482e-a963-62add79dbccd","Type":"ContainerStarted","Data":"01a368ab2e7e955695503f57ac879470317a1baf26cbc65b2d77da3d2cc08994"}
Sep 10 14:23:49.408055 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:49.408001    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hr9qp" podStartSLOduration=1.800305168 podStartE2EDuration="3.407968263s" totalImagesPullingTime="1.607663095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:23:49.407287657 +0000 UTC m=+247.598815880" watchObservedRunningTime="2026-09-10 14:23:49.407968263 +0000 UTC m=+247.599496460"
Sep 10 14:23:50.675670 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.675636    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"]
Sep 10 14:23:50.678080 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.678063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"
Sep 10 14:23:50.680548 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.680525    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 14:23:50.680669 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.680531    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-clm7d\""
Sep 10 14:23:50.681506 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.681492    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 14:23:50.689107 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.689083    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"]
Sep 10 14:23:50.790248 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.790214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d14845f-55d3-493b-b689-5740c4b2ad4a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w\" (UID: \"7d14845f-55d3-493b-b689-5740c4b2ad4a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"
Sep 10 14:23:50.790415 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.790282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d14845f-55d3-493b-b689-5740c4b2ad4a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w\" (UID: \"7d14845f-55d3-493b-b689-5740c4b2ad4a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"
Sep 10 14:23:50.790415 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.790305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lfrh\" (UniqueName: \"kubernetes.io/projected/7d14845f-55d3-493b-b689-5740c4b2ad4a-kube-api-access-6lfrh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w\" (UID: \"7d14845f-55d3-493b-b689-5740c4b2ad4a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"
Sep 10 14:23:50.891133 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.891099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d14845f-55d3-493b-b689-5740c4b2ad4a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w\" (UID: \"7d14845f-55d3-493b-b689-5740c4b2ad4a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"
Sep 10 14:23:50.891284 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.891138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lfrh\" (UniqueName: \"kubernetes.io/projected/7d14845f-55d3-493b-b689-5740c4b2ad4a-kube-api-access-6lfrh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w\" (UID: \"7d14845f-55d3-493b-b689-5740c4b2ad4a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"
Sep 10 14:23:50.891284 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.891201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d14845f-55d3-493b-b689-5740c4b2ad4a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w\" (UID: \"7d14845f-55d3-493b-b689-5740c4b2ad4a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"
Sep 10 14:23:50.891488 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.891468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d14845f-55d3-493b-b689-5740c4b2ad4a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w\" (UID: \"7d14845f-55d3-493b-b689-5740c4b2ad4a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"
Sep 10 14:23:50.891577 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.891546    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d14845f-55d3-493b-b689-5740c4b2ad4a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w\" (UID: \"7d14845f-55d3-493b-b689-5740c4b2ad4a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"
Sep 10 14:23:50.899797 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.899779    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lfrh\" (UniqueName: \"kubernetes.io/projected/7d14845f-55d3-493b-b689-5740c4b2ad4a-kube-api-access-6lfrh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w\" (UID: \"7d14845f-55d3-493b-b689-5740c4b2ad4a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"
Sep 10 14:23:50.986508 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:50.986418    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"
Sep 10 14:23:51.255557 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:51.255473    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"]
Sep 10 14:23:51.258288 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:23:51.258260    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d14845f_55d3_493b_b689_5740c4b2ad4a.slice/crio-0e508490d772511689e678151374488fd9cc226d81b2075911409861b13b6406 WatchSource:0}: Error finding container 0e508490d772511689e678151374488fd9cc226d81b2075911409861b13b6406: Status 404 returned error can't find the container with id 0e508490d772511689e678151374488fd9cc226d81b2075911409861b13b6406
Sep 10 14:23:51.379214 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:51.379178    2580 generic.go:356] "Generic (PLEG): container finished" podID="7d14845f-55d3-493b-b689-5740c4b2ad4a" containerID="9488d5d74ba82a9f3dead0e01fb83aca7e2b2d82c7d740ccdc9c6341e399c64f" exitCode=0
Sep 10 14:23:51.379344 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:51.379222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w" event={"ID":"7d14845f-55d3-493b-b689-5740c4b2ad4a","Type":"ContainerDied","Data":"9488d5d74ba82a9f3dead0e01fb83aca7e2b2d82c7d740ccdc9c6341e399c64f"}
Sep 10 14:23:51.379344 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:51.379266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w" event={"ID":"7d14845f-55d3-493b-b689-5740c4b2ad4a","Type":"ContainerStarted","Data":"0e508490d772511689e678151374488fd9cc226d81b2075911409861b13b6406"}
Sep 10 14:23:52.688305 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:52.688273    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9pt25"]
Sep 10 14:23:52.690436 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:52.690421    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pt25"
Sep 10 14:23:52.693695 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:52.693668    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 14:23:52.694646 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:52.694621    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-pq7rg\""
Sep 10 14:23:52.694737 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:52.694630    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 14:23:52.703564 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:52.703539    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9pt25"]
Sep 10 14:23:52.807417 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:52.807376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2883a8f-d1f0-41ac-a2a8-36852f22e17c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9pt25\" (UID: \"b2883a8f-d1f0-41ac-a2a8-36852f22e17c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pt25"
Sep 10 14:23:52.807614 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:52.807491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9kkj\" (UniqueName: \"kubernetes.io/projected/b2883a8f-d1f0-41ac-a2a8-36852f22e17c-kube-api-access-f9kkj\") pod \"cert-manager-cainjector-559cdd8c87-9pt25\" (UID: \"b2883a8f-d1f0-41ac-a2a8-36852f22e17c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pt25"
Sep 10 14:23:52.908635 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:52.908598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9kkj\" (UniqueName: \"kubernetes.io/projected/b2883a8f-d1f0-41ac-a2a8-36852f22e17c-kube-api-access-f9kkj\") pod \"cert-manager-cainjector-559cdd8c87-9pt25\" (UID: \"b2883a8f-d1f0-41ac-a2a8-36852f22e17c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pt25"
Sep 10 14:23:52.908809 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:52.908717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2883a8f-d1f0-41ac-a2a8-36852f22e17c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9pt25\" (UID: \"b2883a8f-d1f0-41ac-a2a8-36852f22e17c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pt25"
Sep 10 14:23:52.916840 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:52.916812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2883a8f-d1f0-41ac-a2a8-36852f22e17c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9pt25\" (UID: \"b2883a8f-d1f0-41ac-a2a8-36852f22e17c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pt25"
Sep 10 14:23:52.917058 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:52.917037    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9kkj\" (UniqueName: \"kubernetes.io/projected/b2883a8f-d1f0-41ac-a2a8-36852f22e17c-kube-api-access-f9kkj\") pod \"cert-manager-cainjector-559cdd8c87-9pt25\" (UID: \"b2883a8f-d1f0-41ac-a2a8-36852f22e17c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pt25"
Sep 10 14:23:53.005941 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:53.005870    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pt25"
Sep 10 14:23:53.627879 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:53.627659    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9pt25"]
Sep 10 14:23:53.630352 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:23:53.630328    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2883a8f_d1f0_41ac_a2a8_36852f22e17c.slice/crio-079df4fd685156ed9729d53dcd17e38380cd33a4593ee8c2a252e6c2e20b9f4d WatchSource:0}: Error finding container 079df4fd685156ed9729d53dcd17e38380cd33a4593ee8c2a252e6c2e20b9f4d: Status 404 returned error can't find the container with id 079df4fd685156ed9729d53dcd17e38380cd33a4593ee8c2a252e6c2e20b9f4d
Sep 10 14:23:54.394359 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:54.394323    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pt25" event={"ID":"b2883a8f-d1f0-41ac-a2a8-36852f22e17c","Type":"ContainerStarted","Data":"079df4fd685156ed9729d53dcd17e38380cd33a4593ee8c2a252e6c2e20b9f4d"}
Sep 10 14:23:54.396037 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:54.396010    2580 generic.go:356] "Generic (PLEG): container finished" podID="7d14845f-55d3-493b-b689-5740c4b2ad4a" containerID="797f27400412d4e42fd076ad048313d2313ced3da1edbb256f46f452dabec046" exitCode=0
Sep 10 14:23:54.396171 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:54.396081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w" event={"ID":"7d14845f-55d3-493b-b689-5740c4b2ad4a","Type":"ContainerDied","Data":"797f27400412d4e42fd076ad048313d2313ced3da1edbb256f46f452dabec046"}
Sep 10 14:23:55.402558 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:55.402526    2580 generic.go:356] "Generic (PLEG): container finished" podID="7d14845f-55d3-493b-b689-5740c4b2ad4a" containerID="c42c72f14d596089b652c55b2afeda6fadefce8f85e26f9cecec0353e0904adb" exitCode=0
Sep 10 14:23:55.402919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:55.402571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w" event={"ID":"7d14845f-55d3-493b-b689-5740c4b2ad4a","Type":"ContainerDied","Data":"c42c72f14d596089b652c55b2afeda6fadefce8f85e26f9cecec0353e0904adb"}
Sep 10 14:23:56.549308 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:56.549279    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"
Sep 10 14:23:56.642462 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:56.642423    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lfrh\" (UniqueName: \"kubernetes.io/projected/7d14845f-55d3-493b-b689-5740c4b2ad4a-kube-api-access-6lfrh\") pod \"7d14845f-55d3-493b-b689-5740c4b2ad4a\" (UID: \"7d14845f-55d3-493b-b689-5740c4b2ad4a\") "
Sep 10 14:23:56.642624 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:56.642481    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d14845f-55d3-493b-b689-5740c4b2ad4a-util\") pod \"7d14845f-55d3-493b-b689-5740c4b2ad4a\" (UID: \"7d14845f-55d3-493b-b689-5740c4b2ad4a\") "
Sep 10 14:23:56.642624 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:56.642513    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d14845f-55d3-493b-b689-5740c4b2ad4a-bundle\") pod \"7d14845f-55d3-493b-b689-5740c4b2ad4a\" (UID: \"7d14845f-55d3-493b-b689-5740c4b2ad4a\") "
Sep 10 14:23:56.642988 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:56.642940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d14845f-55d3-493b-b689-5740c4b2ad4a-bundle" (OuterVolumeSpecName: "bundle") pod "7d14845f-55d3-493b-b689-5740c4b2ad4a" (UID: "7d14845f-55d3-493b-b689-5740c4b2ad4a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:23:56.644901 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:56.644865    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d14845f-55d3-493b-b689-5740c4b2ad4a-kube-api-access-6lfrh" (OuterVolumeSpecName: "kube-api-access-6lfrh") pod "7d14845f-55d3-493b-b689-5740c4b2ad4a" (UID: "7d14845f-55d3-493b-b689-5740c4b2ad4a"). InnerVolumeSpecName "kube-api-access-6lfrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:23:56.650938 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:56.650912    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d14845f-55d3-493b-b689-5740c4b2ad4a-util" (OuterVolumeSpecName: "util") pod "7d14845f-55d3-493b-b689-5740c4b2ad4a" (UID: "7d14845f-55d3-493b-b689-5740c4b2ad4a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:23:56.743261 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:56.743184    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lfrh\" (UniqueName: \"kubernetes.io/projected/7d14845f-55d3-493b-b689-5740c4b2ad4a-kube-api-access-6lfrh\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:23:56.743261 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:56.743219    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d14845f-55d3-493b-b689-5740c4b2ad4a-util\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:23:56.743261 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:56.743235    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d14845f-55d3-493b-b689-5740c4b2ad4a-bundle\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:23:57.415523 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:57.415493    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w" event={"ID":"7d14845f-55d3-493b-b689-5740c4b2ad4a","Type":"ContainerDied","Data":"0e508490d772511689e678151374488fd9cc226d81b2075911409861b13b6406"}
Sep 10 14:23:57.415709 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:57.415539    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e508490d772511689e678151374488fd9cc226d81b2075911409861b13b6406"
Sep 10 14:23:57.415709 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:57.415540    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzpm9w"
Sep 10 14:23:58.422475 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:58.422440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pt25" event={"ID":"b2883a8f-d1f0-41ac-a2a8-36852f22e17c","Type":"ContainerStarted","Data":"1cd80cb87932d9db2ce52f87b6f9092e9c4d90ae85e286b0b4448b0c4a964090"}
Sep 10 14:23:58.438186 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:23:58.438124    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pt25" podStartSLOduration=1.7475981200000001 podStartE2EDuration="6.438101127s" totalImagesPullingTime="4.690503007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:23:58.437043964 +0000 UTC m=+256.628572164" watchObservedRunningTime="2026-09-10 14:23:58.438101127 +0000 UTC m=+256.629629325"
Sep 10 14:24:00.530602 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.530563    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lt2qs"]
Sep 10 14:24:00.531091 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.530868    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d14845f-55d3-493b-b689-5740c4b2ad4a" containerName="pull"
Sep 10 14:24:00.531091 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.530879    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d14845f-55d3-493b-b689-5740c4b2ad4a" containerName="pull"
Sep 10 14:24:00.531091 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.530897    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d14845f-55d3-493b-b689-5740c4b2ad4a" containerName="util"
Sep 10 14:24:00.531091 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.530902    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d14845f-55d3-493b-b689-5740c4b2ad4a" containerName="util"
Sep 10 14:24:00.531091 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.530909    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d14845f-55d3-493b-b689-5740c4b2ad4a" containerName="extract"
Sep 10 14:24:00.531091 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.530916    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d14845f-55d3-493b-b689-5740c4b2ad4a" containerName="extract"
Sep 10 14:24:00.531091 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.530965    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d14845f-55d3-493b-b689-5740c4b2ad4a" containerName="extract"
Sep 10 14:24:00.532801 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.532785    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lt2qs"
Sep 10 14:24:00.535314 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.535292    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qx85l\""
Sep 10 14:24:00.543792 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.543771    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lt2qs"]
Sep 10 14:24:00.677132 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.677089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19c6861c-1d99-43be-a2ec-6accde28f7bf-bound-sa-token\") pod \"cert-manager-55b4d954d5-lt2qs\" (UID: \"19c6861c-1d99-43be-a2ec-6accde28f7bf\") " pod="cert-manager/cert-manager-55b4d954d5-lt2qs"
Sep 10 14:24:00.677295 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.677141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmft2\" (UniqueName: \"kubernetes.io/projected/19c6861c-1d99-43be-a2ec-6accde28f7bf-kube-api-access-jmft2\") pod \"cert-manager-55b4d954d5-lt2qs\" (UID: \"19c6861c-1d99-43be-a2ec-6accde28f7bf\") " pod="cert-manager/cert-manager-55b4d954d5-lt2qs"
Sep 10 14:24:00.778523 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.778491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19c6861c-1d99-43be-a2ec-6accde28f7bf-bound-sa-token\") pod \"cert-manager-55b4d954d5-lt2qs\" (UID: \"19c6861c-1d99-43be-a2ec-6accde28f7bf\") " pod="cert-manager/cert-manager-55b4d954d5-lt2qs"
Sep 10 14:24:00.778682 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.778535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmft2\" (UniqueName: \"kubernetes.io/projected/19c6861c-1d99-43be-a2ec-6accde28f7bf-kube-api-access-jmft2\") pod \"cert-manager-55b4d954d5-lt2qs\" (UID: \"19c6861c-1d99-43be-a2ec-6accde28f7bf\") " pod="cert-manager/cert-manager-55b4d954d5-lt2qs"
Sep 10 14:24:00.787724 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.787661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19c6861c-1d99-43be-a2ec-6accde28f7bf-bound-sa-token\") pod \"cert-manager-55b4d954d5-lt2qs\" (UID: \"19c6861c-1d99-43be-a2ec-6accde28f7bf\") " pod="cert-manager/cert-manager-55b4d954d5-lt2qs"
Sep 10 14:24:00.787955 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.787934    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmft2\" (UniqueName: \"kubernetes.io/projected/19c6861c-1d99-43be-a2ec-6accde28f7bf-kube-api-access-jmft2\") pod \"cert-manager-55b4d954d5-lt2qs\" (UID: \"19c6861c-1d99-43be-a2ec-6accde28f7bf\") " pod="cert-manager/cert-manager-55b4d954d5-lt2qs"
Sep 10 14:24:00.841774 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:00.841739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lt2qs"
Sep 10 14:24:01.108950 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:01.108928    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lt2qs"]
Sep 10 14:24:01.111117 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:24:01.111088    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19c6861c_1d99_43be_a2ec_6accde28f7bf.slice/crio-a205ccd8d8664daeee63046ef7be9fc91625cdb7267f6d59372f203737ca1030 WatchSource:0}: Error finding container a205ccd8d8664daeee63046ef7be9fc91625cdb7267f6d59372f203737ca1030: Status 404 returned error can't find the container with id a205ccd8d8664daeee63046ef7be9fc91625cdb7267f6d59372f203737ca1030
Sep 10 14:24:01.436611 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:01.436578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lt2qs" event={"ID":"19c6861c-1d99-43be-a2ec-6accde28f7bf","Type":"ContainerStarted","Data":"7f9df8fe951e05b3c4b7f3df48b99d7dccb9409515545c011863821e5374bb33"}
Sep 10 14:24:01.436756 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:01.436616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lt2qs" event={"ID":"19c6861c-1d99-43be-a2ec-6accde28f7bf","Type":"ContainerStarted","Data":"a205ccd8d8664daeee63046ef7be9fc91625cdb7267f6d59372f203737ca1030"}
Sep 10 14:24:01.453179 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:01.453123    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-lt2qs" podStartSLOduration=1.453105033 podStartE2EDuration="1.453105033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:01.451719276 +0000 UTC m=+259.643247474" watchObservedRunningTime="2026-09-10 14:24:01.453105033 +0000 UTC m=+259.644633232"
Sep 10 14:24:07.338676 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.338601    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"]
Sep 10 14:24:07.342171 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.342155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"
Sep 10 14:24:07.345112 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.345091    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 14:24:07.345243 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.345147    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 14:24:07.346143 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.346128    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-clm7d\""
Sep 10 14:24:07.351699 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.351678    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"]
Sep 10 14:24:07.434838 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.434808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdlr7\" (UniqueName: \"kubernetes.io/projected/0412f3c6-1e11-42e2-877e-1f384135b50e-kube-api-access-vdlr7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx\" (UID: \"0412f3c6-1e11-42e2-877e-1f384135b50e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"
Sep 10 14:24:07.435037 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.434848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0412f3c6-1e11-42e2-877e-1f384135b50e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx\" (UID: \"0412f3c6-1e11-42e2-877e-1f384135b50e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"
Sep 10 14:24:07.435037 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.434955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0412f3c6-1e11-42e2-877e-1f384135b50e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx\" (UID: \"0412f3c6-1e11-42e2-877e-1f384135b50e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"
Sep 10 14:24:07.536005 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.535948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0412f3c6-1e11-42e2-877e-1f384135b50e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx\" (UID: \"0412f3c6-1e11-42e2-877e-1f384135b50e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"
Sep 10 14:24:07.536175 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.536079    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0412f3c6-1e11-42e2-877e-1f384135b50e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx\" (UID: \"0412f3c6-1e11-42e2-877e-1f384135b50e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"
Sep 10 14:24:07.536175 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.536144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdlr7\" (UniqueName: \"kubernetes.io/projected/0412f3c6-1e11-42e2-877e-1f384135b50e-kube-api-access-vdlr7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx\" (UID: \"0412f3c6-1e11-42e2-877e-1f384135b50e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"
Sep 10 14:24:07.536340 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.536311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0412f3c6-1e11-42e2-877e-1f384135b50e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx\" (UID: \"0412f3c6-1e11-42e2-877e-1f384135b50e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"
Sep 10 14:24:07.536420 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.536396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0412f3c6-1e11-42e2-877e-1f384135b50e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx\" (UID: \"0412f3c6-1e11-42e2-877e-1f384135b50e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"
Sep 10 14:24:07.544796 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.544769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdlr7\" (UniqueName: \"kubernetes.io/projected/0412f3c6-1e11-42e2-877e-1f384135b50e-kube-api-access-vdlr7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx\" (UID: \"0412f3c6-1e11-42e2-877e-1f384135b50e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"
Sep 10 14:24:07.653372 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.653343    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"
Sep 10 14:24:07.921891 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:07.921866    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"]
Sep 10 14:24:07.924068 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:24:07.924042    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0412f3c6_1e11_42e2_877e_1f384135b50e.slice/crio-750ec63b954b6196e2d294560fadd084ff643ab3b1c224bd05d3b89efbfb2a57 WatchSource:0}: Error finding container 750ec63b954b6196e2d294560fadd084ff643ab3b1c224bd05d3b89efbfb2a57: Status 404 returned error can't find the container with id 750ec63b954b6196e2d294560fadd084ff643ab3b1c224bd05d3b89efbfb2a57
Sep 10 14:24:08.469339 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:08.469305    2580 generic.go:356] "Generic (PLEG): container finished" podID="0412f3c6-1e11-42e2-877e-1f384135b50e" containerID="ddd09e1c32ae05dfffa1f1307353a0d0837bdb80cb676823d2fd35cdb234a681" exitCode=0
Sep 10 14:24:08.469720 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:08.469388    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx" event={"ID":"0412f3c6-1e11-42e2-877e-1f384135b50e","Type":"ContainerDied","Data":"ddd09e1c32ae05dfffa1f1307353a0d0837bdb80cb676823d2fd35cdb234a681"}
Sep 10 14:24:08.469720 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:08.469435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx" event={"ID":"0412f3c6-1e11-42e2-877e-1f384135b50e","Type":"ContainerStarted","Data":"750ec63b954b6196e2d294560fadd084ff643ab3b1c224bd05d3b89efbfb2a57"}
Sep 10 14:24:09.476337 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:09.476299    2580 generic.go:356] "Generic (PLEG): container finished" podID="0412f3c6-1e11-42e2-877e-1f384135b50e" containerID="06a7ae0834419e4ecee73520a9239066e0f6ff880099b70870f2d54cf2d6772e" exitCode=0
Sep 10 14:24:09.476692 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:09.476381    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx" event={"ID":"0412f3c6-1e11-42e2-877e-1f384135b50e","Type":"ContainerDied","Data":"06a7ae0834419e4ecee73520a9239066e0f6ff880099b70870f2d54cf2d6772e"}
Sep 10 14:24:10.483420 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:10.483388    2580 generic.go:356] "Generic (PLEG): container finished" podID="0412f3c6-1e11-42e2-877e-1f384135b50e" containerID="69ee7d15233362a349b31a294143b73582b9f3eaf8ff50e0ced5888f90c1b60c" exitCode=0
Sep 10 14:24:10.483804 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:10.483427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx" event={"ID":"0412f3c6-1e11-42e2-877e-1f384135b50e","Type":"ContainerDied","Data":"69ee7d15233362a349b31a294143b73582b9f3eaf8ff50e0ced5888f90c1b60c"}
Sep 10 14:24:11.634003 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:11.633958    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"
Sep 10 14:24:11.669422 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:11.669392    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdlr7\" (UniqueName: \"kubernetes.io/projected/0412f3c6-1e11-42e2-877e-1f384135b50e-kube-api-access-vdlr7\") pod \"0412f3c6-1e11-42e2-877e-1f384135b50e\" (UID: \"0412f3c6-1e11-42e2-877e-1f384135b50e\") "
Sep 10 14:24:11.669572 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:11.669435    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0412f3c6-1e11-42e2-877e-1f384135b50e-util\") pod \"0412f3c6-1e11-42e2-877e-1f384135b50e\" (UID: \"0412f3c6-1e11-42e2-877e-1f384135b50e\") "
Sep 10 14:24:11.669572 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:11.669504    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0412f3c6-1e11-42e2-877e-1f384135b50e-bundle\") pod \"0412f3c6-1e11-42e2-877e-1f384135b50e\" (UID: \"0412f3c6-1e11-42e2-877e-1f384135b50e\") "
Sep 10 14:24:11.670196 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:11.670170    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0412f3c6-1e11-42e2-877e-1f384135b50e-bundle" (OuterVolumeSpecName: "bundle") pod "0412f3c6-1e11-42e2-877e-1f384135b50e" (UID: "0412f3c6-1e11-42e2-877e-1f384135b50e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:24:11.671440 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:11.671416    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0412f3c6-1e11-42e2-877e-1f384135b50e-kube-api-access-vdlr7" (OuterVolumeSpecName: "kube-api-access-vdlr7") pod "0412f3c6-1e11-42e2-877e-1f384135b50e" (UID: "0412f3c6-1e11-42e2-877e-1f384135b50e"). InnerVolumeSpecName "kube-api-access-vdlr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:24:11.674773 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:11.674747    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0412f3c6-1e11-42e2-877e-1f384135b50e-util" (OuterVolumeSpecName: "util") pod "0412f3c6-1e11-42e2-877e-1f384135b50e" (UID: "0412f3c6-1e11-42e2-877e-1f384135b50e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:24:11.770787 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:11.770707    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdlr7\" (UniqueName: \"kubernetes.io/projected/0412f3c6-1e11-42e2-877e-1f384135b50e-kube-api-access-vdlr7\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:24:11.770787 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:11.770734    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0412f3c6-1e11-42e2-877e-1f384135b50e-util\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:24:11.770787 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:11.770744    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0412f3c6-1e11-42e2-877e-1f384135b50e-bundle\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:24:12.493951 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:12.493920    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx"
Sep 10 14:24:12.493951 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:12.493933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2hsx" event={"ID":"0412f3c6-1e11-42e2-877e-1f384135b50e","Type":"ContainerDied","Data":"750ec63b954b6196e2d294560fadd084ff643ab3b1c224bd05d3b89efbfb2a57"}
Sep 10 14:24:12.494199 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:12.493997    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="750ec63b954b6196e2d294560fadd084ff643ab3b1c224bd05d3b89efbfb2a57"
Sep 10 14:24:22.662982 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.662949    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"]
Sep 10 14:24:22.663334 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.663257    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0412f3c6-1e11-42e2-877e-1f384135b50e" containerName="extract"
Sep 10 14:24:22.663334 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.663269    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0412f3c6-1e11-42e2-877e-1f384135b50e" containerName="extract"
Sep 10 14:24:22.663334 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.663279    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0412f3c6-1e11-42e2-877e-1f384135b50e" containerName="util"
Sep 10 14:24:22.663334 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.663284    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0412f3c6-1e11-42e2-877e-1f384135b50e" containerName="util"
Sep 10 14:24:22.663334 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.663294    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0412f3c6-1e11-42e2-877e-1f384135b50e" containerName="pull"
Sep 10 14:24:22.663334 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.663299    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0412f3c6-1e11-42e2-877e-1f384135b50e" containerName="pull"
Sep 10 14:24:22.663530 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.663359    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0412f3c6-1e11-42e2-877e-1f384135b50e" containerName="extract"
Sep 10 14:24:22.665941 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.665923    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"
Sep 10 14:24:22.668449 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.668420    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 14:24:22.668566 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.668430    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 14:24:22.669491 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.669469    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-clm7d\""
Sep 10 14:24:22.674530 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.674506    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"]
Sep 10 14:24:22.765586 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.765551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f030242-f5c6-46ac-87ee-23a6c89c63b5-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv\" (UID: \"8f030242-f5c6-46ac-87ee-23a6c89c63b5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"
Sep 10 14:24:22.765754 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.765617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njbth\" (UniqueName: \"kubernetes.io/projected/8f030242-f5c6-46ac-87ee-23a6c89c63b5-kube-api-access-njbth\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv\" (UID: \"8f030242-f5c6-46ac-87ee-23a6c89c63b5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"
Sep 10 14:24:22.765754 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.765643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f030242-f5c6-46ac-87ee-23a6c89c63b5-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv\" (UID: \"8f030242-f5c6-46ac-87ee-23a6c89c63b5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"
Sep 10 14:24:22.866740 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.866705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f030242-f5c6-46ac-87ee-23a6c89c63b5-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv\" (UID: \"8f030242-f5c6-46ac-87ee-23a6c89c63b5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"
Sep 10 14:24:22.866912 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.866769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njbth\" (UniqueName: \"kubernetes.io/projected/8f030242-f5c6-46ac-87ee-23a6c89c63b5-kube-api-access-njbth\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv\" (UID: \"8f030242-f5c6-46ac-87ee-23a6c89c63b5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"
Sep 10 14:24:22.866912 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.866793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f030242-f5c6-46ac-87ee-23a6c89c63b5-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv\" (UID: \"8f030242-f5c6-46ac-87ee-23a6c89c63b5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"
Sep 10 14:24:22.867104 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.867085    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f030242-f5c6-46ac-87ee-23a6c89c63b5-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv\" (UID: \"8f030242-f5c6-46ac-87ee-23a6c89c63b5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"
Sep 10 14:24:22.867164 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.867110    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f030242-f5c6-46ac-87ee-23a6c89c63b5-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv\" (UID: \"8f030242-f5c6-46ac-87ee-23a6c89c63b5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"
Sep 10 14:24:22.881627 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.881594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njbth\" (UniqueName: \"kubernetes.io/projected/8f030242-f5c6-46ac-87ee-23a6c89c63b5-kube-api-access-njbth\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv\" (UID: \"8f030242-f5c6-46ac-87ee-23a6c89c63b5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"
Sep 10 14:24:22.975135 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:22.975060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"
Sep 10 14:24:23.253346 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.253324    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"]
Sep 10 14:24:23.255458 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:24:23.255429    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f030242_f5c6_46ac_87ee_23a6c89c63b5.slice/crio-7f24ca6c4368756b9030380931bff193abcc57dbdc9a043912ab3c9b4670f9a9 WatchSource:0}: Error finding container 7f24ca6c4368756b9030380931bff193abcc57dbdc9a043912ab3c9b4670f9a9: Status 404 returned error can't find the container with id 7f24ca6c4368756b9030380931bff193abcc57dbdc9a043912ab3c9b4670f9a9
Sep 10 14:24:23.544432 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.544335    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f030242-f5c6-46ac-87ee-23a6c89c63b5" containerID="6fb9c5911186e2a876e988f47e76db5dcacebc80978fc85a744384a34b7a6bbc" exitCode=0
Sep 10 14:24:23.544563 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.544429    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv" event={"ID":"8f030242-f5c6-46ac-87ee-23a6c89c63b5","Type":"ContainerDied","Data":"6fb9c5911186e2a876e988f47e76db5dcacebc80978fc85a744384a34b7a6bbc"}
Sep 10 14:24:23.544563 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.544477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv" event={"ID":"8f030242-f5c6-46ac-87ee-23a6c89c63b5","Type":"ContainerStarted","Data":"7f24ca6c4368756b9030380931bff193abcc57dbdc9a043912ab3c9b4670f9a9"}
Sep 10 14:24:23.924371 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.924338    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"]
Sep 10 14:24:23.927901 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.927884    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:23.930551 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.930530    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 14:24:23.930949 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.930933    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:24:23.931646 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.931626    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 14:24:23.931793 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.931678    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 14:24:23.931793 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.931684    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-x4xjl\""
Sep 10 14:24:23.931947 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.931848    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 14:24:23.937067 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.937046    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"]
Sep 10 14:24:23.977196 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.977162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0d93290-18b4-49c9-b462-8c8c219ddf77-cert\") pod \"lws-controller-manager-759b87488b-n8mk6\" (UID: \"a0d93290-18b4-49c9-b462-8c8c219ddf77\") " pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:23.977387 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.977201    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a0d93290-18b4-49c9-b462-8c8c219ddf77-manager-config\") pod \"lws-controller-manager-759b87488b-n8mk6\" (UID: \"a0d93290-18b4-49c9-b462-8c8c219ddf77\") " pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:23.977387 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.977239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbzcx\" (UniqueName: \"kubernetes.io/projected/a0d93290-18b4-49c9-b462-8c8c219ddf77-kube-api-access-lbzcx\") pod \"lws-controller-manager-759b87488b-n8mk6\" (UID: \"a0d93290-18b4-49c9-b462-8c8c219ddf77\") " pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:23.977387 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:23.977307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0d93290-18b4-49c9-b462-8c8c219ddf77-metrics-cert\") pod \"lws-controller-manager-759b87488b-n8mk6\" (UID: \"a0d93290-18b4-49c9-b462-8c8c219ddf77\") " pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:24.078226 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.078176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbzcx\" (UniqueName: \"kubernetes.io/projected/a0d93290-18b4-49c9-b462-8c8c219ddf77-kube-api-access-lbzcx\") pod \"lws-controller-manager-759b87488b-n8mk6\" (UID: \"a0d93290-18b4-49c9-b462-8c8c219ddf77\") " pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:24.078226 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.078229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0d93290-18b4-49c9-b462-8c8c219ddf77-metrics-cert\") pod \"lws-controller-manager-759b87488b-n8mk6\" (UID: \"a0d93290-18b4-49c9-b462-8c8c219ddf77\") " pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:24.078498 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.078285    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0d93290-18b4-49c9-b462-8c8c219ddf77-cert\") pod \"lws-controller-manager-759b87488b-n8mk6\" (UID: \"a0d93290-18b4-49c9-b462-8c8c219ddf77\") " pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:24.078498 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.078308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a0d93290-18b4-49c9-b462-8c8c219ddf77-manager-config\") pod \"lws-controller-manager-759b87488b-n8mk6\" (UID: \"a0d93290-18b4-49c9-b462-8c8c219ddf77\") " pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:24.078966 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.078941    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a0d93290-18b4-49c9-b462-8c8c219ddf77-manager-config\") pod \"lws-controller-manager-759b87488b-n8mk6\" (UID: \"a0d93290-18b4-49c9-b462-8c8c219ddf77\") " pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:24.080846 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.080822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0d93290-18b4-49c9-b462-8c8c219ddf77-metrics-cert\") pod \"lws-controller-manager-759b87488b-n8mk6\" (UID: \"a0d93290-18b4-49c9-b462-8c8c219ddf77\") " pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:24.080941 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.080894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0d93290-18b4-49c9-b462-8c8c219ddf77-cert\") pod \"lws-controller-manager-759b87488b-n8mk6\" (UID: \"a0d93290-18b4-49c9-b462-8c8c219ddf77\") " pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:24.088604 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.088581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbzcx\" (UniqueName: \"kubernetes.io/projected/a0d93290-18b4-49c9-b462-8c8c219ddf77-kube-api-access-lbzcx\") pod \"lws-controller-manager-759b87488b-n8mk6\" (UID: \"a0d93290-18b4-49c9-b462-8c8c219ddf77\") " pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:24.168309 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.168230    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"]
Sep 10 14:24:24.172097 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.172077    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.174661 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.174643    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p5jmf\""
Sep 10 14:24:24.175196 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.175182    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 14:24:24.175787 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.175766    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 14:24:24.175866 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.175766    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 14:24:24.175866 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.175801    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 14:24:24.189564 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.189541    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"]
Sep 10 14:24:24.237958 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.237926    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:24.279581 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.279503    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/69a116ca-0e8a-413a-9e47-3d78f6d18441-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-drn2t\" (UID: \"69a116ca-0e8a-413a-9e47-3d78f6d18441\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.279581 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.279560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/69a116ca-0e8a-413a-9e47-3d78f6d18441-manifests\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-drn2t\" (UID: \"69a116ca-0e8a-413a-9e47-3d78f6d18441\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.279581 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.279587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69a116ca-0e8a-413a-9e47-3d78f6d18441-tmp\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-drn2t\" (UID: \"69a116ca-0e8a-413a-9e47-3d78f6d18441\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.279581 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.279613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69a116ca-0e8a-413a-9e47-3d78f6d18441-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-drn2t\" (UID: \"69a116ca-0e8a-413a-9e47-3d78f6d18441\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.280179 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.279649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dzrz\" (UniqueName: \"kubernetes.io/projected/69a116ca-0e8a-413a-9e47-3d78f6d18441-kube-api-access-2dzrz\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-drn2t\" (UID: \"69a116ca-0e8a-413a-9e47-3d78f6d18441\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.380882 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.380844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dzrz\" (UniqueName: \"kubernetes.io/projected/69a116ca-0e8a-413a-9e47-3d78f6d18441-kube-api-access-2dzrz\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-drn2t\" (UID: \"69a116ca-0e8a-413a-9e47-3d78f6d18441\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.381076 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.380961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/69a116ca-0e8a-413a-9e47-3d78f6d18441-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-drn2t\" (UID: \"69a116ca-0e8a-413a-9e47-3d78f6d18441\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.381076 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.381016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/69a116ca-0e8a-413a-9e47-3d78f6d18441-manifests\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-drn2t\" (UID: \"69a116ca-0e8a-413a-9e47-3d78f6d18441\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.381076 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.381041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69a116ca-0e8a-413a-9e47-3d78f6d18441-tmp\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-drn2t\" (UID: \"69a116ca-0e8a-413a-9e47-3d78f6d18441\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.381076 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.381066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69a116ca-0e8a-413a-9e47-3d78f6d18441-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-drn2t\" (UID: \"69a116ca-0e8a-413a-9e47-3d78f6d18441\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.381445 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.381418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/69a116ca-0e8a-413a-9e47-3d78f6d18441-manifests\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-drn2t\" (UID: \"69a116ca-0e8a-413a-9e47-3d78f6d18441\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.383398 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.383372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69a116ca-0e8a-413a-9e47-3d78f6d18441-tmp\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-drn2t\" (UID: \"69a116ca-0e8a-413a-9e47-3d78f6d18441\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.383550 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.383534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69a116ca-0e8a-413a-9e47-3d78f6d18441-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-drn2t\" (UID: \"69a116ca-0e8a-413a-9e47-3d78f6d18441\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.383602 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.383585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/69a116ca-0e8a-413a-9e47-3d78f6d18441-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-drn2t\" (UID: \"69a116ca-0e8a-413a-9e47-3d78f6d18441\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.388857 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.388837    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dzrz\" (UniqueName: \"kubernetes.io/projected/69a116ca-0e8a-413a-9e47-3d78f6d18441-kube-api-access-2dzrz\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-drn2t\" (UID: \"69a116ca-0e8a-413a-9e47-3d78f6d18441\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.482057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.482033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:24.528798 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.528719    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"]
Sep 10 14:24:24.532905 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:24:24.532875    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d93290_18b4_49c9_b462_8c8c219ddf77.slice/crio-e7fc4dcc555df1fe3b70ffb2f2e4acb54f25d788b7d81afd13b371813127413e WatchSource:0}: Error finding container e7fc4dcc555df1fe3b70ffb2f2e4acb54f25d788b7d81afd13b371813127413e: Status 404 returned error can't find the container with id e7fc4dcc555df1fe3b70ffb2f2e4acb54f25d788b7d81afd13b371813127413e
Sep 10 14:24:24.551620 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.551585    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f030242-f5c6-46ac-87ee-23a6c89c63b5" containerID="9896658a7eec89ccfb0e65ee00582b420969008e0af37468a68296dfb68221f5" exitCode=0
Sep 10 14:24:24.551753 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.551671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv" event={"ID":"8f030242-f5c6-46ac-87ee-23a6c89c63b5","Type":"ContainerDied","Data":"9896658a7eec89ccfb0e65ee00582b420969008e0af37468a68296dfb68221f5"}
Sep 10 14:24:24.553288 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.553219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6" event={"ID":"a0d93290-18b4-49c9-b462-8c8c219ddf77","Type":"ContainerStarted","Data":"e7fc4dcc555df1fe3b70ffb2f2e4acb54f25d788b7d81afd13b371813127413e"}
Sep 10 14:24:24.760691 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:24.760663    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"]
Sep 10 14:24:24.764303 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:24:24.764261    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a116ca_0e8a_413a_9e47_3d78f6d18441.slice/crio-698d876d4db2e16ad6f8998b29a05dfe19aaed008679d578ee9671e86651d9f1 WatchSource:0}: Error finding container 698d876d4db2e16ad6f8998b29a05dfe19aaed008679d578ee9671e86651d9f1: Status 404 returned error can't find the container with id 698d876d4db2e16ad6f8998b29a05dfe19aaed008679d578ee9671e86651d9f1
Sep 10 14:24:25.561069 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:25.560962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t" event={"ID":"69a116ca-0e8a-413a-9e47-3d78f6d18441","Type":"ContainerStarted","Data":"698d876d4db2e16ad6f8998b29a05dfe19aaed008679d578ee9671e86651d9f1"}
Sep 10 14:24:25.564233 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:25.564005    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f030242-f5c6-46ac-87ee-23a6c89c63b5" containerID="34a5ee4dc1aa0058d0b1b563e9883262c7e36988b95c1cb34ec08026ab7c89ab" exitCode=0
Sep 10 14:24:25.564233 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:25.564128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv" event={"ID":"8f030242-f5c6-46ac-87ee-23a6c89c63b5","Type":"ContainerDied","Data":"34a5ee4dc1aa0058d0b1b563e9883262c7e36988b95c1cb34ec08026ab7c89ab"}
Sep 10 14:24:27.225016 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:27.224989    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"
Sep 10 14:24:27.307417 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:27.307381    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njbth\" (UniqueName: \"kubernetes.io/projected/8f030242-f5c6-46ac-87ee-23a6c89c63b5-kube-api-access-njbth\") pod \"8f030242-f5c6-46ac-87ee-23a6c89c63b5\" (UID: \"8f030242-f5c6-46ac-87ee-23a6c89c63b5\") "
Sep 10 14:24:27.307598 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:27.307434    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f030242-f5c6-46ac-87ee-23a6c89c63b5-util\") pod \"8f030242-f5c6-46ac-87ee-23a6c89c63b5\" (UID: \"8f030242-f5c6-46ac-87ee-23a6c89c63b5\") "
Sep 10 14:24:27.307598 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:27.307499    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f030242-f5c6-46ac-87ee-23a6c89c63b5-bundle\") pod \"8f030242-f5c6-46ac-87ee-23a6c89c63b5\" (UID: \"8f030242-f5c6-46ac-87ee-23a6c89c63b5\") "
Sep 10 14:24:27.308419 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:27.308382    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f030242-f5c6-46ac-87ee-23a6c89c63b5-bundle" (OuterVolumeSpecName: "bundle") pod "8f030242-f5c6-46ac-87ee-23a6c89c63b5" (UID: "8f030242-f5c6-46ac-87ee-23a6c89c63b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:24:27.310341 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:27.310312    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f030242-f5c6-46ac-87ee-23a6c89c63b5-kube-api-access-njbth" (OuterVolumeSpecName: "kube-api-access-njbth") pod "8f030242-f5c6-46ac-87ee-23a6c89c63b5" (UID: "8f030242-f5c6-46ac-87ee-23a6c89c63b5"). InnerVolumeSpecName "kube-api-access-njbth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:24:27.313176 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:27.313153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f030242-f5c6-46ac-87ee-23a6c89c63b5-util" (OuterVolumeSpecName: "util") pod "8f030242-f5c6-46ac-87ee-23a6c89c63b5" (UID: "8f030242-f5c6-46ac-87ee-23a6c89c63b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:24:27.408757 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:27.408708    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njbth\" (UniqueName: \"kubernetes.io/projected/8f030242-f5c6-46ac-87ee-23a6c89c63b5-kube-api-access-njbth\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:24:27.408757 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:27.408750    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f030242-f5c6-46ac-87ee-23a6c89c63b5-util\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:24:27.408757 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:27.408764    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f030242-f5c6-46ac-87ee-23a6c89c63b5-bundle\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:24:27.577236 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:27.577147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv" event={"ID":"8f030242-f5c6-46ac-87ee-23a6c89c63b5","Type":"ContainerDied","Data":"7f24ca6c4368756b9030380931bff193abcc57dbdc9a043912ab3c9b4670f9a9"}
Sep 10 14:24:27.577236 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:27.577173    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c8qtv"
Sep 10 14:24:27.577236 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:27.577206    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f24ca6c4368756b9030380931bff193abcc57dbdc9a043912ab3c9b4670f9a9"
Sep 10 14:24:28.584329 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:28.584289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6" event={"ID":"a0d93290-18b4-49c9-b462-8c8c219ddf77","Type":"ContainerStarted","Data":"2a760b53ae5b2b9e5167bac01fa4b30f5ec8cca0660ba43b9d568dcc06a03d8e"}
Sep 10 14:24:28.584732 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:28.584352    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:28.585879 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:28.585848    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t" event={"ID":"69a116ca-0e8a-413a-9e47-3d78f6d18441","Type":"ContainerStarted","Data":"daf01d382efb67f8a0d0843c9a4310b8bcc592ef6d5354d8cb8ef066102e5062"}
Sep 10 14:24:28.605061 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:28.605007    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6" podStartSLOduration=2.030813885 podStartE2EDuration="5.604991212s" totalImagesPullingTime="3.574177327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:28.603783638 +0000 UTC m=+286.795311835" watchObservedRunningTime="2026-09-10 14:24:28.604991212 +0000 UTC m=+286.796519410"
Sep 10 14:24:29.592295 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:29.592205    2580 generic.go:356] "Generic (PLEG): container finished" podID="69a116ca-0e8a-413a-9e47-3d78f6d18441" containerID="daf01d382efb67f8a0d0843c9a4310b8bcc592ef6d5354d8cb8ef066102e5062" exitCode=0
Sep 10 14:24:29.592667 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:29.592294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t" event={"ID":"69a116ca-0e8a-413a-9e47-3d78f6d18441","Type":"ContainerDied","Data":"daf01d382efb67f8a0d0843c9a4310b8bcc592ef6d5354d8cb8ef066102e5062"}
Sep 10 14:24:30.598564 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:30.598532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t" event={"ID":"69a116ca-0e8a-413a-9e47-3d78f6d18441","Type":"ContainerStarted","Data":"bb24864a87592d74c2b43f4ba3d7003c00b1db95a7ec47d33bdfa4a38884722e"}
Sep 10 14:24:30.598941 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:30.598668    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:30.622225 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:30.622172    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t" podStartSLOduration=1.680824893 podStartE2EDuration="6.622158342s" totalImagesPullingTime="3.807764098s" totalInitContainerRuntime="1.133569351s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:30.620151059 +0000 UTC m=+288.811679292" watchObservedRunningTime="2026-09-10 14:24:30.622158342 +0000 UTC m=+288.813686539"
Sep 10 14:24:39.595449 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:39.595413    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-759b87488b-n8mk6"
Sep 10 14:24:41.604892 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:41.604863    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-drn2t"
Sep 10 14:24:42.210342 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:42.210313    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:24:42.211170 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:42.211143    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:24:42.217839 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:42.217820    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 14:24:44.311458 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.311423    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf"]
Sep 10 14:24:44.312687 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.311889    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f030242-f5c6-46ac-87ee-23a6c89c63b5" containerName="util"
Sep 10 14:24:44.312687 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.311903    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f030242-f5c6-46ac-87ee-23a6c89c63b5" containerName="util"
Sep 10 14:24:44.312687 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.311915    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f030242-f5c6-46ac-87ee-23a6c89c63b5" containerName="pull"
Sep 10 14:24:44.312687 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.311924    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f030242-f5c6-46ac-87ee-23a6c89c63b5" containerName="pull"
Sep 10 14:24:44.312687 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.311953    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f030242-f5c6-46ac-87ee-23a6c89c63b5" containerName="extract"
Sep 10 14:24:44.312687 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.311961    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f030242-f5c6-46ac-87ee-23a6c89c63b5" containerName="extract"
Sep 10 14:24:44.312687 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.312054    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f030242-f5c6-46ac-87ee-23a6c89c63b5" containerName="extract"
Sep 10 14:24:44.314117 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.314100    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf"
Sep 10 14:24:44.318042 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.318016    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 14:24:44.318180 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.318096    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 14:24:44.318180 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.318116    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kg5c6\""
Sep 10 14:24:44.318319 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.318300    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 14:24:44.321078 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.320837    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 14:24:44.336586 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.336563    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf"]
Sep 10 14:24:44.453847 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.453812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84dxz\" (UniqueName: \"kubernetes.io/projected/6f820d02-66ad-4454-9b97-84fb994bbf71-kube-api-access-84dxz\") pod \"kube-auth-proxy-79c7cbc6bf-jxtdf\" (UID: \"6f820d02-66ad-4454-9b97-84fb994bbf71\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf"
Sep 10 14:24:44.454042 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.453861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f820d02-66ad-4454-9b97-84fb994bbf71-tmp\") pod \"kube-auth-proxy-79c7cbc6bf-jxtdf\" (UID: \"6f820d02-66ad-4454-9b97-84fb994bbf71\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf"
Sep 10 14:24:44.454042 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.453941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f820d02-66ad-4454-9b97-84fb994bbf71-tls-certs\") pod \"kube-auth-proxy-79c7cbc6bf-jxtdf\" (UID: \"6f820d02-66ad-4454-9b97-84fb994bbf71\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf"
Sep 10 14:24:44.554606 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.554568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84dxz\" (UniqueName: \"kubernetes.io/projected/6f820d02-66ad-4454-9b97-84fb994bbf71-kube-api-access-84dxz\") pod \"kube-auth-proxy-79c7cbc6bf-jxtdf\" (UID: \"6f820d02-66ad-4454-9b97-84fb994bbf71\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf"
Sep 10 14:24:44.554767 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.554623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f820d02-66ad-4454-9b97-84fb994bbf71-tmp\") pod \"kube-auth-proxy-79c7cbc6bf-jxtdf\" (UID: \"6f820d02-66ad-4454-9b97-84fb994bbf71\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf"
Sep 10 14:24:44.554767 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.554666    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f820d02-66ad-4454-9b97-84fb994bbf71-tls-certs\") pod \"kube-auth-proxy-79c7cbc6bf-jxtdf\" (UID: \"6f820d02-66ad-4454-9b97-84fb994bbf71\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf"
Sep 10 14:24:44.556901 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.556868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f820d02-66ad-4454-9b97-84fb994bbf71-tmp\") pod \"kube-auth-proxy-79c7cbc6bf-jxtdf\" (UID: \"6f820d02-66ad-4454-9b97-84fb994bbf71\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf"
Sep 10 14:24:44.557053 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.557021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f820d02-66ad-4454-9b97-84fb994bbf71-tls-certs\") pod \"kube-auth-proxy-79c7cbc6bf-jxtdf\" (UID: \"6f820d02-66ad-4454-9b97-84fb994bbf71\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf"
Sep 10 14:24:44.564433 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.564370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84dxz\" (UniqueName: \"kubernetes.io/projected/6f820d02-66ad-4454-9b97-84fb994bbf71-kube-api-access-84dxz\") pod \"kube-auth-proxy-79c7cbc6bf-jxtdf\" (UID: \"6f820d02-66ad-4454-9b97-84fb994bbf71\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf"
Sep 10 14:24:44.623765 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.623740    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf"
Sep 10 14:24:44.901808 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.901776    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf"]
Sep 10 14:24:44.904853 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:24:44.904824    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f820d02_66ad_4454_9b97_84fb994bbf71.slice/crio-7b8a59cedb0948eb655cc9d6d99461582a7b421162e76a44cbdf60d00f45ec74 WatchSource:0}: Error finding container 7b8a59cedb0948eb655cc9d6d99461582a7b421162e76a44cbdf60d00f45ec74: Status 404 returned error can't find the container with id 7b8a59cedb0948eb655cc9d6d99461582a7b421162e76a44cbdf60d00f45ec74
Sep 10 14:24:44.906802 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:44.906784    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:24:45.676758 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:45.676715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf" event={"ID":"6f820d02-66ad-4454-9b97-84fb994bbf71","Type":"ContainerStarted","Data":"7b8a59cedb0948eb655cc9d6d99461582a7b421162e76a44cbdf60d00f45ec74"}
Sep 10 14:24:48.695342 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:48.695298    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf" event={"ID":"6f820d02-66ad-4454-9b97-84fb994bbf71","Type":"ContainerStarted","Data":"73e1f8496d3a1544d081a2a0e075309163cefac592949cb77d7baa6860c50e75"}
Sep 10 14:24:48.712964 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:24:48.712906    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-jxtdf" podStartSLOduration=1.720132808 podStartE2EDuration="4.712889993s" totalImagesPullingTime="2.992757185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:48.711074228 +0000 UTC m=+306.902602426" watchObservedRunningTime="2026-09-10 14:24:48.712889993 +0000 UTC m=+306.904418190"
Sep 10 14:26:00.192471 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.192387    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"]
Sep 10 14:26:00.195206 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.195179    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"
Sep 10 14:26:00.197916 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.197894    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-clm7d\""
Sep 10 14:26:00.198042 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.197922    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 14:26:00.199034 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.199015    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 14:26:00.204582 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.204555    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"]
Sep 10 14:26:00.292819 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.292786    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"]
Sep 10 14:26:00.295198 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.295181    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"
Sep 10 14:26:00.295815 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.295792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlf58\" (UniqueName: \"kubernetes.io/projected/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-kube-api-access-mlf58\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n\" (UID: \"ad4ed0f5-8b0e-4da8-bfdc-07090510546b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"
Sep 10 14:26:00.295914 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.295835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n\" (UID: \"ad4ed0f5-8b0e-4da8-bfdc-07090510546b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"
Sep 10 14:26:00.295993 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.295909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n\" (UID: \"ad4ed0f5-8b0e-4da8-bfdc-07090510546b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"
Sep 10 14:26:00.310293 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.310269    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"]
Sep 10 14:26:00.388338 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.388299    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"]
Sep 10 14:26:00.390726 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.390710    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"
Sep 10 14:26:00.396655 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.396625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsn47\" (UniqueName: \"kubernetes.io/projected/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-kube-api-access-bsn47\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms\" (UID: \"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"
Sep 10 14:26:00.396798 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.396675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms\" (UID: \"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"
Sep 10 14:26:00.396798 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.396718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlf58\" (UniqueName: \"kubernetes.io/projected/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-kube-api-access-mlf58\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n\" (UID: \"ad4ed0f5-8b0e-4da8-bfdc-07090510546b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"
Sep 10 14:26:00.396798 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.396760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n\" (UID: \"ad4ed0f5-8b0e-4da8-bfdc-07090510546b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"
Sep 10 14:26:00.396798 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.396787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms\" (UID: \"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"
Sep 10 14:26:00.397135 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.396824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n\" (UID: \"ad4ed0f5-8b0e-4da8-bfdc-07090510546b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"
Sep 10 14:26:00.397463 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.397439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n\" (UID: \"ad4ed0f5-8b0e-4da8-bfdc-07090510546b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"
Sep 10 14:26:00.397509 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.397460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n\" (UID: \"ad4ed0f5-8b0e-4da8-bfdc-07090510546b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"
Sep 10 14:26:00.401345 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.401324    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"]
Sep 10 14:26:00.405854 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.405823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlf58\" (UniqueName: \"kubernetes.io/projected/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-kube-api-access-mlf58\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n\" (UID: \"ad4ed0f5-8b0e-4da8-bfdc-07090510546b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"
Sep 10 14:26:00.488594 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.488517    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"]
Sep 10 14:26:00.491076 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.491059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"
Sep 10 14:26:00.497515 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.497476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/383e45db-3f67-458c-b06c-132f4877ee06-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj\" (UID: \"383e45db-3f67-458c-b06c-132f4877ee06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"
Sep 10 14:26:00.497669 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.497537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms\" (UID: \"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"
Sep 10 14:26:00.497669 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.497569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfp4t\" (UniqueName: \"kubernetes.io/projected/383e45db-3f67-458c-b06c-132f4877ee06-kube-api-access-mfp4t\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj\" (UID: \"383e45db-3f67-458c-b06c-132f4877ee06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"
Sep 10 14:26:00.497891 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.497668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsn47\" (UniqueName: \"kubernetes.io/projected/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-kube-api-access-bsn47\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms\" (UID: \"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"
Sep 10 14:26:00.497891 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.497704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/383e45db-3f67-458c-b06c-132f4877ee06-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj\" (UID: \"383e45db-3f67-458c-b06c-132f4877ee06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"
Sep 10 14:26:00.497891 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.497760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms\" (UID: \"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"
Sep 10 14:26:00.498072 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.497984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms\" (UID: \"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"
Sep 10 14:26:00.498126 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.498076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms\" (UID: \"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"
Sep 10 14:26:00.500674 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.500652    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"]
Sep 10 14:26:00.504579 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.504559    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"
Sep 10 14:26:00.505635 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.505609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsn47\" (UniqueName: \"kubernetes.io/projected/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-kube-api-access-bsn47\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms\" (UID: \"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"
Sep 10 14:26:00.598654 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.598618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/447d7faa-1ea9-4410-b40c-64e6a1245bde-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g\" (UID: \"447d7faa-1ea9-4410-b40c-64e6a1245bde\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"
Sep 10 14:26:00.598835 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.598668    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/447d7faa-1ea9-4410-b40c-64e6a1245bde-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g\" (UID: \"447d7faa-1ea9-4410-b40c-64e6a1245bde\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"
Sep 10 14:26:00.598835 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.598692    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/383e45db-3f67-458c-b06c-132f4877ee06-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj\" (UID: \"383e45db-3f67-458c-b06c-132f4877ee06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"
Sep 10 14:26:00.598835 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.598741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw5vc\" (UniqueName: \"kubernetes.io/projected/447d7faa-1ea9-4410-b40c-64e6a1245bde-kube-api-access-sw5vc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g\" (UID: \"447d7faa-1ea9-4410-b40c-64e6a1245bde\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"
Sep 10 14:26:00.599028 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.598844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/383e45db-3f67-458c-b06c-132f4877ee06-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj\" (UID: \"383e45db-3f67-458c-b06c-132f4877ee06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"
Sep 10 14:26:00.599028 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.598885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfp4t\" (UniqueName: \"kubernetes.io/projected/383e45db-3f67-458c-b06c-132f4877ee06-kube-api-access-mfp4t\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj\" (UID: \"383e45db-3f67-458c-b06c-132f4877ee06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"
Sep 10 14:26:00.599128 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.599080    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/383e45db-3f67-458c-b06c-132f4877ee06-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj\" (UID: \"383e45db-3f67-458c-b06c-132f4877ee06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"
Sep 10 14:26:00.599128 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.599119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/383e45db-3f67-458c-b06c-132f4877ee06-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj\" (UID: \"383e45db-3f67-458c-b06c-132f4877ee06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"
Sep 10 14:26:00.604899 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.604879    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"
Sep 10 14:26:00.607019 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.606996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfp4t\" (UniqueName: \"kubernetes.io/projected/383e45db-3f67-458c-b06c-132f4877ee06-kube-api-access-mfp4t\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj\" (UID: \"383e45db-3f67-458c-b06c-132f4877ee06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"
Sep 10 14:26:00.699806 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.699766    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"
Sep 10 14:26:00.700095 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.700073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/447d7faa-1ea9-4410-b40c-64e6a1245bde-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g\" (UID: \"447d7faa-1ea9-4410-b40c-64e6a1245bde\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"
Sep 10 14:26:00.700161 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.700113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/447d7faa-1ea9-4410-b40c-64e6a1245bde-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g\" (UID: \"447d7faa-1ea9-4410-b40c-64e6a1245bde\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"
Sep 10 14:26:00.700161 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.700139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw5vc\" (UniqueName: \"kubernetes.io/projected/447d7faa-1ea9-4410-b40c-64e6a1245bde-kube-api-access-sw5vc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g\" (UID: \"447d7faa-1ea9-4410-b40c-64e6a1245bde\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"
Sep 10 14:26:00.700545 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.700525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/447d7faa-1ea9-4410-b40c-64e6a1245bde-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g\" (UID: \"447d7faa-1ea9-4410-b40c-64e6a1245bde\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"
Sep 10 14:26:00.700545 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.700537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/447d7faa-1ea9-4410-b40c-64e6a1245bde-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g\" (UID: \"447d7faa-1ea9-4410-b40c-64e6a1245bde\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"
Sep 10 14:26:00.708870 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.708841    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw5vc\" (UniqueName: \"kubernetes.io/projected/447d7faa-1ea9-4410-b40c-64e6a1245bde-kube-api-access-sw5vc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g\" (UID: \"447d7faa-1ea9-4410-b40c-64e6a1245bde\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"
Sep 10 14:26:00.775530 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.775507    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"]
Sep 10 14:26:00.777766 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:26:00.777739    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4ed0f5_8b0e_4da8_bfdc_07090510546b.slice/crio-638bde3dd62969fe627b6537c80cfda8e93ec9368e31d2ed0173771d61321136 WatchSource:0}: Error finding container 638bde3dd62969fe627b6537c80cfda8e93ec9368e31d2ed0173771d61321136: Status 404 returned error can't find the container with id 638bde3dd62969fe627b6537c80cfda8e93ec9368e31d2ed0173771d61321136
Sep 10 14:26:00.801321 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.801288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"
Sep 10 14:26:00.879250 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.878524    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"]
Sep 10 14:26:00.880361 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:26:00.880331    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a2157b_e5a3_4781_8e19_71f7f3c8bd2f.slice/crio-8996025af147f3187f3447d3174544848b1872a55fbe22ce79aeed494d0032ab WatchSource:0}: Error finding container 8996025af147f3187f3447d3174544848b1872a55fbe22ce79aeed494d0032ab: Status 404 returned error can't find the container with id 8996025af147f3187f3447d3174544848b1872a55fbe22ce79aeed494d0032ab
Sep 10 14:26:00.971995 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:00.971943    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"]
Sep 10 14:26:00.972394 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:26:00.972374    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383e45db_3f67_458c_b06c_132f4877ee06.slice/crio-2e5e8224ba0e36cfa819111ba1d6c6b6a7e2b3e416fd9fe2b155c452986a4106 WatchSource:0}: Error finding container 2e5e8224ba0e36cfa819111ba1d6c6b6a7e2b3e416fd9fe2b155c452986a4106: Status 404 returned error can't find the container with id 2e5e8224ba0e36cfa819111ba1d6c6b6a7e2b3e416fd9fe2b155c452986a4106
Sep 10 14:26:01.030105 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:01.030076    2580 generic.go:356] "Generic (PLEG): container finished" podID="45a2157b-e5a3-4781-8e19-71f7f3c8bd2f" containerID="d950fbaf11c326c065b734d32fba45aa5dca1288b21713a05f782b0820986520" exitCode=0
Sep 10 14:26:01.030210 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:01.030159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms" event={"ID":"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f","Type":"ContainerDied","Data":"d950fbaf11c326c065b734d32fba45aa5dca1288b21713a05f782b0820986520"}
Sep 10 14:26:01.030210 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:01.030207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms" event={"ID":"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f","Type":"ContainerStarted","Data":"8996025af147f3187f3447d3174544848b1872a55fbe22ce79aeed494d0032ab"}
Sep 10 14:26:01.031399 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:01.031373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj" event={"ID":"383e45db-3f67-458c-b06c-132f4877ee06","Type":"ContainerStarted","Data":"2e5e8224ba0e36cfa819111ba1d6c6b6a7e2b3e416fd9fe2b155c452986a4106"}
Sep 10 14:26:01.032885 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:01.032863    2580 generic.go:356] "Generic (PLEG): container finished" podID="ad4ed0f5-8b0e-4da8-bfdc-07090510546b" containerID="de491090df0dfa9e063d91fc834fa43540052d12852b69f30a56d808327164a1" exitCode=0
Sep 10 14:26:01.033002 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:01.032937    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n" event={"ID":"ad4ed0f5-8b0e-4da8-bfdc-07090510546b","Type":"ContainerDied","Data":"de491090df0dfa9e063d91fc834fa43540052d12852b69f30a56d808327164a1"}
Sep 10 14:26:01.033002 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:01.032989    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n" event={"ID":"ad4ed0f5-8b0e-4da8-bfdc-07090510546b","Type":"ContainerStarted","Data":"638bde3dd62969fe627b6537c80cfda8e93ec9368e31d2ed0173771d61321136"}
Sep 10 14:26:01.076104 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:01.076081    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"]
Sep 10 14:26:01.078001 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:26:01.077960    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447d7faa_1ea9_4410_b40c_64e6a1245bde.slice/crio-45dec14a7f64b70db6b8fe1a7933e2a73dfec7bb4462b81d6a8b09c5ee91e1b7 WatchSource:0}: Error finding container 45dec14a7f64b70db6b8fe1a7933e2a73dfec7bb4462b81d6a8b09c5ee91e1b7: Status 404 returned error can't find the container with id 45dec14a7f64b70db6b8fe1a7933e2a73dfec7bb4462b81d6a8b09c5ee91e1b7
Sep 10 14:26:02.043380 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:02.043351    2580 generic.go:356] "Generic (PLEG): container finished" podID="ad4ed0f5-8b0e-4da8-bfdc-07090510546b" containerID="2ec12f33a015511aec4ba6caff3cf88c5e45b5f7280d5602b84f3b8f3575040a" exitCode=0
Sep 10 14:26:02.043780 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:02.043431    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n" event={"ID":"ad4ed0f5-8b0e-4da8-bfdc-07090510546b","Type":"ContainerDied","Data":"2ec12f33a015511aec4ba6caff3cf88c5e45b5f7280d5602b84f3b8f3575040a"}
Sep 10 14:26:02.045435 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:02.045416    2580 generic.go:356] "Generic (PLEG): container finished" podID="45a2157b-e5a3-4781-8e19-71f7f3c8bd2f" containerID="828a3eadfc31268dcbbe9f7ce04b89e21d327080b277201aa336194a4400e1fd" exitCode=0
Sep 10 14:26:02.045519 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:02.045495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms" event={"ID":"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f","Type":"ContainerDied","Data":"828a3eadfc31268dcbbe9f7ce04b89e21d327080b277201aa336194a4400e1fd"}
Sep 10 14:26:02.047043 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:02.047021    2580 generic.go:356] "Generic (PLEG): container finished" podID="383e45db-3f67-458c-b06c-132f4877ee06" containerID="a08c4085da36953717a03d48fcaa86b724f4a47f010b5c389a159935e70cf500" exitCode=0
Sep 10 14:26:02.047165 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:02.047141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj" event={"ID":"383e45db-3f67-458c-b06c-132f4877ee06","Type":"ContainerDied","Data":"a08c4085da36953717a03d48fcaa86b724f4a47f010b5c389a159935e70cf500"}
Sep 10 14:26:02.048932 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:02.048911    2580 generic.go:356] "Generic (PLEG): container finished" podID="447d7faa-1ea9-4410-b40c-64e6a1245bde" containerID="4a9fe09e91dc7f720b71950ed29ab2f7067a9a60280519a6616cb1ea9b142ad2" exitCode=0
Sep 10 14:26:02.049034 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:02.048946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g" event={"ID":"447d7faa-1ea9-4410-b40c-64e6a1245bde","Type":"ContainerDied","Data":"4a9fe09e91dc7f720b71950ed29ab2f7067a9a60280519a6616cb1ea9b142ad2"}
Sep 10 14:26:02.049034 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:02.049008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g" event={"ID":"447d7faa-1ea9-4410-b40c-64e6a1245bde","Type":"ContainerStarted","Data":"45dec14a7f64b70db6b8fe1a7933e2a73dfec7bb4462b81d6a8b09c5ee91e1b7"}
Sep 10 14:26:03.058059 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:03.058029    2580 generic.go:356] "Generic (PLEG): container finished" podID="ad4ed0f5-8b0e-4da8-bfdc-07090510546b" containerID="044bbb324785625c8d932079132f0ff5b0eb1f607fa4fa63aa2246e04b31f185" exitCode=0
Sep 10 14:26:03.058428 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:03.058084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n" event={"ID":"ad4ed0f5-8b0e-4da8-bfdc-07090510546b","Type":"ContainerDied","Data":"044bbb324785625c8d932079132f0ff5b0eb1f607fa4fa63aa2246e04b31f185"}
Sep 10 14:26:03.060317 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:03.060296    2580 generic.go:356] "Generic (PLEG): container finished" podID="45a2157b-e5a3-4781-8e19-71f7f3c8bd2f" containerID="e676ce05711745dd8820f8f6ac399a9b1e6f70e556db50473ed240781bc03bc0" exitCode=0
Sep 10 14:26:03.060438 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:03.060381    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms" event={"ID":"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f","Type":"ContainerDied","Data":"e676ce05711745dd8820f8f6ac399a9b1e6f70e556db50473ed240781bc03bc0"}
Sep 10 14:26:03.062376 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:03.062353    2580 generic.go:356] "Generic (PLEG): container finished" podID="447d7faa-1ea9-4410-b40c-64e6a1245bde" containerID="5c43f1d68300120beac7acd1c1e8d820a27cf6b6a4ecd0240843d0ea8a5ae9a1" exitCode=0
Sep 10 14:26:03.062462 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:03.062390    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g" event={"ID":"447d7faa-1ea9-4410-b40c-64e6a1245bde","Type":"ContainerDied","Data":"5c43f1d68300120beac7acd1c1e8d820a27cf6b6a4ecd0240843d0ea8a5ae9a1"}
Sep 10 14:26:04.070154 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.070116    2580 generic.go:356] "Generic (PLEG): container finished" podID="447d7faa-1ea9-4410-b40c-64e6a1245bde" containerID="78799070414d04dd9d1219317623f2f40c60828d2324d9a864a0e13abe97f7d9" exitCode=0
Sep 10 14:26:04.070551 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.070202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g" event={"ID":"447d7faa-1ea9-4410-b40c-64e6a1245bde","Type":"ContainerDied","Data":"78799070414d04dd9d1219317623f2f40c60828d2324d9a864a0e13abe97f7d9"}
Sep 10 14:26:04.072150 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.072126    2580 generic.go:356] "Generic (PLEG): container finished" podID="383e45db-3f67-458c-b06c-132f4877ee06" containerID="0669316e39025832242f72232832d0fdd0dc8251e0c0da69de8f39d2415624b1" exitCode=0
Sep 10 14:26:04.072269 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.072205    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj" event={"ID":"383e45db-3f67-458c-b06c-132f4877ee06","Type":"ContainerDied","Data":"0669316e39025832242f72232832d0fdd0dc8251e0c0da69de8f39d2415624b1"}
Sep 10 14:26:04.286412 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.286385    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"
Sep 10 14:26:04.289176 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.289156    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"
Sep 10 14:26:04.330396 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.330332    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-bundle\") pod \"ad4ed0f5-8b0e-4da8-bfdc-07090510546b\" (UID: \"ad4ed0f5-8b0e-4da8-bfdc-07090510546b\") "
Sep 10 14:26:04.330396 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.330373    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-util\") pod \"ad4ed0f5-8b0e-4da8-bfdc-07090510546b\" (UID: \"ad4ed0f5-8b0e-4da8-bfdc-07090510546b\") "
Sep 10 14:26:04.330601 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.330413    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsn47\" (UniqueName: \"kubernetes.io/projected/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-kube-api-access-bsn47\") pod \"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f\" (UID: \"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f\") "
Sep 10 14:26:04.330601 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.330429    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-bundle\") pod \"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f\" (UID: \"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f\") "
Sep 10 14:26:04.330601 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.330447    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-util\") pod \"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f\" (UID: \"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f\") "
Sep 10 14:26:04.330601 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.330496    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlf58\" (UniqueName: \"kubernetes.io/projected/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-kube-api-access-mlf58\") pod \"ad4ed0f5-8b0e-4da8-bfdc-07090510546b\" (UID: \"ad4ed0f5-8b0e-4da8-bfdc-07090510546b\") "
Sep 10 14:26:04.331280 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.330903    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-bundle" (OuterVolumeSpecName: "bundle") pod "ad4ed0f5-8b0e-4da8-bfdc-07090510546b" (UID: "ad4ed0f5-8b0e-4da8-bfdc-07090510546b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:26:04.331492 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.331470    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-bundle" (OuterVolumeSpecName: "bundle") pod "45a2157b-e5a3-4781-8e19-71f7f3c8bd2f" (UID: "45a2157b-e5a3-4781-8e19-71f7f3c8bd2f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:26:04.333278 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.333226    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-kube-api-access-bsn47" (OuterVolumeSpecName: "kube-api-access-bsn47") pod "45a2157b-e5a3-4781-8e19-71f7f3c8bd2f" (UID: "45a2157b-e5a3-4781-8e19-71f7f3c8bd2f"). InnerVolumeSpecName "kube-api-access-bsn47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:26:04.333387 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.333339    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-kube-api-access-mlf58" (OuterVolumeSpecName: "kube-api-access-mlf58") pod "ad4ed0f5-8b0e-4da8-bfdc-07090510546b" (UID: "ad4ed0f5-8b0e-4da8-bfdc-07090510546b"). InnerVolumeSpecName "kube-api-access-mlf58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:26:04.336657 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.336629    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-util" (OuterVolumeSpecName: "util") pod "45a2157b-e5a3-4781-8e19-71f7f3c8bd2f" (UID: "45a2157b-e5a3-4781-8e19-71f7f3c8bd2f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:26:04.337466 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.337443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-util" (OuterVolumeSpecName: "util") pod "ad4ed0f5-8b0e-4da8-bfdc-07090510546b" (UID: "ad4ed0f5-8b0e-4da8-bfdc-07090510546b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:26:04.431400 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.431366    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsn47\" (UniqueName: \"kubernetes.io/projected/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-kube-api-access-bsn47\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:26:04.431400 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.431394    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-bundle\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:26:04.431400 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.431404    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45a2157b-e5a3-4781-8e19-71f7f3c8bd2f-util\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:26:04.431609 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.431414    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlf58\" (UniqueName: \"kubernetes.io/projected/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-kube-api-access-mlf58\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:26:04.431609 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.431423    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-bundle\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:26:04.431609 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:04.431431    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad4ed0f5-8b0e-4da8-bfdc-07090510546b-util\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:26:05.078561 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.078534    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n"
Sep 10 14:26:05.078915 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.078559    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njh8n" event={"ID":"ad4ed0f5-8b0e-4da8-bfdc-07090510546b","Type":"ContainerDied","Data":"638bde3dd62969fe627b6537c80cfda8e93ec9368e31d2ed0173771d61321136"}
Sep 10 14:26:05.078915 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.078610    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="638bde3dd62969fe627b6537c80cfda8e93ec9368e31d2ed0173771d61321136"
Sep 10 14:26:05.080537 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.080518    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms"
Sep 10 14:26:05.080652 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.080517    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2bkms" event={"ID":"45a2157b-e5a3-4781-8e19-71f7f3c8bd2f","Type":"ContainerDied","Data":"8996025af147f3187f3447d3174544848b1872a55fbe22ce79aeed494d0032ab"}
Sep 10 14:26:05.080652 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.080632    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8996025af147f3187f3447d3174544848b1872a55fbe22ce79aeed494d0032ab"
Sep 10 14:26:05.082756 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.082734    2580 generic.go:356] "Generic (PLEG): container finished" podID="383e45db-3f67-458c-b06c-132f4877ee06" containerID="607a11c172bfc87c0bca72d94253ec24864e92cc9af0a12a487661db9b457111" exitCode=0
Sep 10 14:26:05.082843 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.082826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj" event={"ID":"383e45db-3f67-458c-b06c-132f4877ee06","Type":"ContainerDied","Data":"607a11c172bfc87c0bca72d94253ec24864e92cc9af0a12a487661db9b457111"}
Sep 10 14:26:05.203913 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.203892    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"
Sep 10 14:26:05.238228 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.238198    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/447d7faa-1ea9-4410-b40c-64e6a1245bde-bundle\") pod \"447d7faa-1ea9-4410-b40c-64e6a1245bde\" (UID: \"447d7faa-1ea9-4410-b40c-64e6a1245bde\") "
Sep 10 14:26:05.238398 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.238282    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/447d7faa-1ea9-4410-b40c-64e6a1245bde-util\") pod \"447d7faa-1ea9-4410-b40c-64e6a1245bde\" (UID: \"447d7faa-1ea9-4410-b40c-64e6a1245bde\") "
Sep 10 14:26:05.238398 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.238348    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw5vc\" (UniqueName: \"kubernetes.io/projected/447d7faa-1ea9-4410-b40c-64e6a1245bde-kube-api-access-sw5vc\") pod \"447d7faa-1ea9-4410-b40c-64e6a1245bde\" (UID: \"447d7faa-1ea9-4410-b40c-64e6a1245bde\") "
Sep 10 14:26:05.238782 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.238748    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447d7faa-1ea9-4410-b40c-64e6a1245bde-bundle" (OuterVolumeSpecName: "bundle") pod "447d7faa-1ea9-4410-b40c-64e6a1245bde" (UID: "447d7faa-1ea9-4410-b40c-64e6a1245bde"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:26:05.240465 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.240439    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447d7faa-1ea9-4410-b40c-64e6a1245bde-kube-api-access-sw5vc" (OuterVolumeSpecName: "kube-api-access-sw5vc") pod "447d7faa-1ea9-4410-b40c-64e6a1245bde" (UID: "447d7faa-1ea9-4410-b40c-64e6a1245bde"). InnerVolumeSpecName "kube-api-access-sw5vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:26:05.243053 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.243031    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447d7faa-1ea9-4410-b40c-64e6a1245bde-util" (OuterVolumeSpecName: "util") pod "447d7faa-1ea9-4410-b40c-64e6a1245bde" (UID: "447d7faa-1ea9-4410-b40c-64e6a1245bde"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:26:05.339540 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.339447    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/447d7faa-1ea9-4410-b40c-64e6a1245bde-util\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:26:05.339540 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.339482    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw5vc\" (UniqueName: \"kubernetes.io/projected/447d7faa-1ea9-4410-b40c-64e6a1245bde-kube-api-access-sw5vc\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:26:05.339540 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:05.339497    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/447d7faa-1ea9-4410-b40c-64e6a1245bde-bundle\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:26:06.090529 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:06.090502    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g"
Sep 10 14:26:06.090907 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:06.090541    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5f7g" event={"ID":"447d7faa-1ea9-4410-b40c-64e6a1245bde","Type":"ContainerDied","Data":"45dec14a7f64b70db6b8fe1a7933e2a73dfec7bb4462b81d6a8b09c5ee91e1b7"}
Sep 10 14:26:06.090907 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:06.090591    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45dec14a7f64b70db6b8fe1a7933e2a73dfec7bb4462b81d6a8b09c5ee91e1b7"
Sep 10 14:26:06.217968 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:06.217944    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"
Sep 10 14:26:06.348561 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:06.348478    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/383e45db-3f67-458c-b06c-132f4877ee06-bundle\") pod \"383e45db-3f67-458c-b06c-132f4877ee06\" (UID: \"383e45db-3f67-458c-b06c-132f4877ee06\") "
Sep 10 14:26:06.348561 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:06.348544    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfp4t\" (UniqueName: \"kubernetes.io/projected/383e45db-3f67-458c-b06c-132f4877ee06-kube-api-access-mfp4t\") pod \"383e45db-3f67-458c-b06c-132f4877ee06\" (UID: \"383e45db-3f67-458c-b06c-132f4877ee06\") "
Sep 10 14:26:06.348820 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:06.348574    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/383e45db-3f67-458c-b06c-132f4877ee06-util\") pod \"383e45db-3f67-458c-b06c-132f4877ee06\" (UID: \"383e45db-3f67-458c-b06c-132f4877ee06\") "
Sep 10 14:26:06.349063 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:06.349034    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383e45db-3f67-458c-b06c-132f4877ee06-bundle" (OuterVolumeSpecName: "bundle") pod "383e45db-3f67-458c-b06c-132f4877ee06" (UID: "383e45db-3f67-458c-b06c-132f4877ee06"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:26:06.349328 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:06.349299    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/383e45db-3f67-458c-b06c-132f4877ee06-bundle\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:26:06.350706 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:06.350681    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383e45db-3f67-458c-b06c-132f4877ee06-kube-api-access-mfp4t" (OuterVolumeSpecName: "kube-api-access-mfp4t") pod "383e45db-3f67-458c-b06c-132f4877ee06" (UID: "383e45db-3f67-458c-b06c-132f4877ee06"). InnerVolumeSpecName "kube-api-access-mfp4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:26:06.393449 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:06.393413    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383e45db-3f67-458c-b06c-132f4877ee06-util" (OuterVolumeSpecName: "util") pod "383e45db-3f67-458c-b06c-132f4877ee06" (UID: "383e45db-3f67-458c-b06c-132f4877ee06"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:26:06.449959 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:06.449925    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfp4t\" (UniqueName: \"kubernetes.io/projected/383e45db-3f67-458c-b06c-132f4877ee06-kube-api-access-mfp4t\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:26:06.449959 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:06.449953    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/383e45db-3f67-458c-b06c-132f4877ee06-util\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:26:07.096843 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:07.096811    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj" event={"ID":"383e45db-3f67-458c-b06c-132f4877ee06","Type":"ContainerDied","Data":"2e5e8224ba0e36cfa819111ba1d6c6b6a7e2b3e416fd9fe2b155c452986a4106"}
Sep 10 14:26:07.096843 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:07.096844    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d766lpj"
Sep 10 14:26:07.096843 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:07.096857    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e5e8224ba0e36cfa819111ba1d6c6b6a7e2b3e416fd9fe2b155c452986a4106"
Sep 10 14:26:29.105752 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.105709    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8"]
Sep 10 14:26:29.106229 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106201    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad4ed0f5-8b0e-4da8-bfdc-07090510546b" containerName="util"
Sep 10 14:26:29.106229 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106219    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad4ed0f5-8b0e-4da8-bfdc-07090510546b" containerName="util"
Sep 10 14:26:29.106366 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106237    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45a2157b-e5a3-4781-8e19-71f7f3c8bd2f" containerName="extract"
Sep 10 14:26:29.106366 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106245    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="45a2157b-e5a3-4781-8e19-71f7f3c8bd2f" containerName="extract"
Sep 10 14:26:29.106366 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106258    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad4ed0f5-8b0e-4da8-bfdc-07090510546b" containerName="pull"
Sep 10 14:26:29.106366 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106266    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad4ed0f5-8b0e-4da8-bfdc-07090510546b" containerName="pull"
Sep 10 14:26:29.106366 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106288    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="447d7faa-1ea9-4410-b40c-64e6a1245bde" containerName="extract"
Sep 10 14:26:29.106366 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106296    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="447d7faa-1ea9-4410-b40c-64e6a1245bde" containerName="extract"
Sep 10 14:26:29.106366 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106306    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383e45db-3f67-458c-b06c-132f4877ee06" containerName="extract"
Sep 10 14:26:29.106366 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106314    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="383e45db-3f67-458c-b06c-132f4877ee06" containerName="extract"
Sep 10 14:26:29.106366 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106327    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad4ed0f5-8b0e-4da8-bfdc-07090510546b" containerName="extract"
Sep 10 14:26:29.106366 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106336    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad4ed0f5-8b0e-4da8-bfdc-07090510546b" containerName="extract"
Sep 10 14:26:29.106366 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106345    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="447d7faa-1ea9-4410-b40c-64e6a1245bde" containerName="pull"
Sep 10 14:26:29.106366 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106352    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="447d7faa-1ea9-4410-b40c-64e6a1245bde" containerName="pull"
Sep 10 14:26:29.106366 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106366    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45a2157b-e5a3-4781-8e19-71f7f3c8bd2f" containerName="util"
Sep 10 14:26:29.106939 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106375    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="45a2157b-e5a3-4781-8e19-71f7f3c8bd2f" containerName="util"
Sep 10 14:26:29.106939 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106388    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383e45db-3f67-458c-b06c-132f4877ee06" containerName="util"
Sep 10 14:26:29.106939 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106396    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="383e45db-3f67-458c-b06c-132f4877ee06" containerName="util"
Sep 10 14:26:29.106939 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106407    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="447d7faa-1ea9-4410-b40c-64e6a1245bde" containerName="util"
Sep 10 14:26:29.106939 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106416    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="447d7faa-1ea9-4410-b40c-64e6a1245bde" containerName="util"
Sep 10 14:26:29.106939 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106429    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45a2157b-e5a3-4781-8e19-71f7f3c8bd2f" containerName="pull"
Sep 10 14:26:29.106939 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106436    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="45a2157b-e5a3-4781-8e19-71f7f3c8bd2f" containerName="pull"
Sep 10 14:26:29.106939 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106445    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383e45db-3f67-458c-b06c-132f4877ee06" containerName="pull"
Sep 10 14:26:29.106939 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106453    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="383e45db-3f67-458c-b06c-132f4877ee06" containerName="pull"
Sep 10 14:26:29.106939 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106536    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="447d7faa-1ea9-4410-b40c-64e6a1245bde" containerName="extract"
Sep 10 14:26:29.106939 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106548    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad4ed0f5-8b0e-4da8-bfdc-07090510546b" containerName="extract"
Sep 10 14:26:29.106939 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106559    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="45a2157b-e5a3-4781-8e19-71f7f3c8bd2f" containerName="extract"
Sep 10 14:26:29.106939 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.106568    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="383e45db-3f67-458c-b06c-132f4877ee06" containerName="extract"
Sep 10 14:26:29.108855 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.108835    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8"
Sep 10 14:26:29.111644 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.111622    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 14:26:29.111644 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.111638    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 14:26:29.112801 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.112779    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 14:26:29.112801 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.112792    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 14:26:29.112933 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.112816    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-drgvk\""
Sep 10 14:26:29.118015 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.117957    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8"]
Sep 10 14:26:29.252332 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.252299    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5ea7525c-71f9-4bc1-875e-61659813fb09-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-cnwx8\" (UID: \"5ea7525c-71f9-4bc1-875e-61659813fb09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8"
Sep 10 14:26:29.252499 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.252342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea7525c-71f9-4bc1-875e-61659813fb09-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-cnwx8\" (UID: \"5ea7525c-71f9-4bc1-875e-61659813fb09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8"
Sep 10 14:26:29.252499 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.252426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7nxz\" (UniqueName: \"kubernetes.io/projected/5ea7525c-71f9-4bc1-875e-61659813fb09-kube-api-access-f7nxz\") pod \"kuadrant-console-plugin-fbdd4f445-cnwx8\" (UID: \"5ea7525c-71f9-4bc1-875e-61659813fb09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8"
Sep 10 14:26:29.353695 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.353652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7nxz\" (UniqueName: \"kubernetes.io/projected/5ea7525c-71f9-4bc1-875e-61659813fb09-kube-api-access-f7nxz\") pod \"kuadrant-console-plugin-fbdd4f445-cnwx8\" (UID: \"5ea7525c-71f9-4bc1-875e-61659813fb09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8"
Sep 10 14:26:29.353850 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.353717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5ea7525c-71f9-4bc1-875e-61659813fb09-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-cnwx8\" (UID: \"5ea7525c-71f9-4bc1-875e-61659813fb09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8"
Sep 10 14:26:29.353850 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.353760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea7525c-71f9-4bc1-875e-61659813fb09-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-cnwx8\" (UID: \"5ea7525c-71f9-4bc1-875e-61659813fb09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8"
Sep 10 14:26:29.353935 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:26:29.353863    2580 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 10 14:26:29.353935 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:26:29.353928    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ea7525c-71f9-4bc1-875e-61659813fb09-plugin-serving-cert podName:5ea7525c-71f9-4bc1-875e-61659813fb09 nodeName:}" failed. No retries permitted until 2026-09-10 14:26:29.853910266 +0000 UTC m=+408.045438444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/5ea7525c-71f9-4bc1-875e-61659813fb09-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-cnwx8" (UID: "5ea7525c-71f9-4bc1-875e-61659813fb09") : secret "plugin-serving-cert" not found
Sep 10 14:26:29.354397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.354376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5ea7525c-71f9-4bc1-875e-61659813fb09-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-cnwx8\" (UID: \"5ea7525c-71f9-4bc1-875e-61659813fb09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8"
Sep 10 14:26:29.362515 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.362453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7nxz\" (UniqueName: \"kubernetes.io/projected/5ea7525c-71f9-4bc1-875e-61659813fb09-kube-api-access-f7nxz\") pod \"kuadrant-console-plugin-fbdd4f445-cnwx8\" (UID: \"5ea7525c-71f9-4bc1-875e-61659813fb09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8"
Sep 10 14:26:29.859690 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.859653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea7525c-71f9-4bc1-875e-61659813fb09-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-cnwx8\" (UID: \"5ea7525c-71f9-4bc1-875e-61659813fb09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8"
Sep 10 14:26:29.862077 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:29.862049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea7525c-71f9-4bc1-875e-61659813fb09-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-cnwx8\" (UID: \"5ea7525c-71f9-4bc1-875e-61659813fb09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8"
Sep 10 14:26:30.019136 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:30.019098    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8"
Sep 10 14:26:30.289116 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:30.289090    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8"]
Sep 10 14:26:30.291211 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:26:30.291181    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ea7525c_71f9_4bc1_875e_61659813fb09.slice/crio-7740ea8261f9fcf871a51497f2e15096ba7a8daa85b3d3055f0d8c694507972b WatchSource:0}: Error finding container 7740ea8261f9fcf871a51497f2e15096ba7a8daa85b3d3055f0d8c694507972b: Status 404 returned error can't find the container with id 7740ea8261f9fcf871a51497f2e15096ba7a8daa85b3d3055f0d8c694507972b
Sep 10 14:26:31.233376 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:31.233337    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8" event={"ID":"5ea7525c-71f9-4bc1-875e-61659813fb09","Type":"ContainerStarted","Data":"7740ea8261f9fcf871a51497f2e15096ba7a8daa85b3d3055f0d8c694507972b"}
Sep 10 14:26:35.261787 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:35.261685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8" event={"ID":"5ea7525c-71f9-4bc1-875e-61659813fb09","Type":"ContainerStarted","Data":"c1b590464f3fdf5364a1e6b028270929fa15cdb281c961790f2fb136afed751d"}
Sep 10 14:26:35.303996 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:35.303935    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cnwx8" podStartSLOduration=1.636231024 podStartE2EDuration="6.303918903s" totalImagesPullingTime="4.667687879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:26:35.3023306 +0000 UTC m=+413.493858799" watchObservedRunningTime="2026-09-10 14:26:35.303918903 +0000 UTC m=+413.495447100"
Sep 10 14:26:40.624383 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:40.624351    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"]
Sep 10 14:26:40.629356 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:40.629331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"
Sep 10 14:26:40.632811 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:40.632787    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zk7bh\""
Sep 10 14:26:40.643727 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:40.643702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2859db2-22aa-4d1a-bc41-db21c90bfe12-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-45jpk\" (UID: \"d2859db2-22aa-4d1a-bc41-db21c90bfe12\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"
Sep 10 14:26:40.643880 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:40.643733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-525jp\" (UniqueName: \"kubernetes.io/projected/d2859db2-22aa-4d1a-bc41-db21c90bfe12-kube-api-access-525jp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-45jpk\" (UID: \"d2859db2-22aa-4d1a-bc41-db21c90bfe12\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"
Sep 10 14:26:40.649328 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:40.649301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"]
Sep 10 14:26:40.745073 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:40.745044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2859db2-22aa-4d1a-bc41-db21c90bfe12-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-45jpk\" (UID: \"d2859db2-22aa-4d1a-bc41-db21c90bfe12\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"
Sep 10 14:26:40.745073 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:40.745075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-525jp\" (UniqueName: \"kubernetes.io/projected/d2859db2-22aa-4d1a-bc41-db21c90bfe12-kube-api-access-525jp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-45jpk\" (UID: \"d2859db2-22aa-4d1a-bc41-db21c90bfe12\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"
Sep 10 14:26:40.745396 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:40.745377    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2859db2-22aa-4d1a-bc41-db21c90bfe12-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-45jpk\" (UID: \"d2859db2-22aa-4d1a-bc41-db21c90bfe12\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"
Sep 10 14:26:40.760746 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:40.760720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-525jp\" (UniqueName: \"kubernetes.io/projected/d2859db2-22aa-4d1a-bc41-db21c90bfe12-kube-api-access-525jp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-45jpk\" (UID: \"d2859db2-22aa-4d1a-bc41-db21c90bfe12\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"
Sep 10 14:26:40.939552 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:40.939515    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"
Sep 10 14:26:41.216717 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:41.216694    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"]
Sep 10 14:26:41.219568 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:26:41.219542    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2859db2_22aa_4d1a_bc41_db21c90bfe12.slice/crio-f073b5b7d7599f488ff8390dda1406e67365175546c1e42f0a3d26d99931b9c1 WatchSource:0}: Error finding container f073b5b7d7599f488ff8390dda1406e67365175546c1e42f0a3d26d99931b9c1: Status 404 returned error can't find the container with id f073b5b7d7599f488ff8390dda1406e67365175546c1e42f0a3d26d99931b9c1
Sep 10 14:26:41.294767 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:41.294739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk" event={"ID":"d2859db2-22aa-4d1a-bc41-db21c90bfe12","Type":"ContainerStarted","Data":"f073b5b7d7599f488ff8390dda1406e67365175546c1e42f0a3d26d99931b9c1"}
Sep 10 14:26:45.323287 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:45.323189    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk" event={"ID":"d2859db2-22aa-4d1a-bc41-db21c90bfe12","Type":"ContainerStarted","Data":"5de987b85e6b5ca922b4ff4315d95a201c02bade8673e122b43c2f9c364660a2"}
Sep 10 14:26:45.323287 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:45.323260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"
Sep 10 14:26:45.347394 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:45.347318    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk" podStartSLOduration=1.5477265089999999 podStartE2EDuration="5.347298121s" totalImagesPullingTime="3.799571612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:26:45.345366083 +0000 UTC m=+423.536894281" watchObservedRunningTime="2026-09-10 14:26:45.347298121 +0000 UTC m=+423.538826319"
Sep 10 14:26:52.617744 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:52.617707    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"]
Sep 10 14:26:52.618184 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:52.617914    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk" podUID="d2859db2-22aa-4d1a-bc41-db21c90bfe12" containerName="manager" containerID="cri-o://5de987b85e6b5ca922b4ff4315d95a201c02bade8673e122b43c2f9c364660a2" gracePeriod=10
Sep 10 14:26:52.619575 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:52.619555    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"
Sep 10 14:26:52.865460 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:52.865433    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"
Sep 10 14:26:52.952358 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:52.952331    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-525jp\" (UniqueName: \"kubernetes.io/projected/d2859db2-22aa-4d1a-bc41-db21c90bfe12-kube-api-access-525jp\") pod \"d2859db2-22aa-4d1a-bc41-db21c90bfe12\" (UID: \"d2859db2-22aa-4d1a-bc41-db21c90bfe12\") "
Sep 10 14:26:52.952508 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:52.952380    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2859db2-22aa-4d1a-bc41-db21c90bfe12-extensions-socket-volume\") pod \"d2859db2-22aa-4d1a-bc41-db21c90bfe12\" (UID: \"d2859db2-22aa-4d1a-bc41-db21c90bfe12\") "
Sep 10 14:26:52.952912 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:52.952879    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2859db2-22aa-4d1a-bc41-db21c90bfe12-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d2859db2-22aa-4d1a-bc41-db21c90bfe12" (UID: "d2859db2-22aa-4d1a-bc41-db21c90bfe12"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:26:52.954609 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:52.954589    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2859db2-22aa-4d1a-bc41-db21c90bfe12-kube-api-access-525jp" (OuterVolumeSpecName: "kube-api-access-525jp") pod "d2859db2-22aa-4d1a-bc41-db21c90bfe12" (UID: "d2859db2-22aa-4d1a-bc41-db21c90bfe12"). InnerVolumeSpecName "kube-api-access-525jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:26:53.053397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:53.053364    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-525jp\" (UniqueName: \"kubernetes.io/projected/d2859db2-22aa-4d1a-bc41-db21c90bfe12-kube-api-access-525jp\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:26:53.053397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:53.053393    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2859db2-22aa-4d1a-bc41-db21c90bfe12-extensions-socket-volume\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:26:53.364891 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:53.364808    2580 generic.go:356] "Generic (PLEG): container finished" podID="d2859db2-22aa-4d1a-bc41-db21c90bfe12" containerID="5de987b85e6b5ca922b4ff4315d95a201c02bade8673e122b43c2f9c364660a2" exitCode=0
Sep 10 14:26:53.364891 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:53.364839    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk" event={"ID":"d2859db2-22aa-4d1a-bc41-db21c90bfe12","Type":"ContainerDied","Data":"5de987b85e6b5ca922b4ff4315d95a201c02bade8673e122b43c2f9c364660a2"}
Sep 10 14:26:53.364891 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:53.364868    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"
Sep 10 14:26:53.365193 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:53.364914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk" event={"ID":"d2859db2-22aa-4d1a-bc41-db21c90bfe12","Type":"ContainerDied","Data":"f073b5b7d7599f488ff8390dda1406e67365175546c1e42f0a3d26d99931b9c1"}
Sep 10 14:26:53.365193 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:53.364944    2580 scope.go:117] "RemoveContainer" containerID="5de987b85e6b5ca922b4ff4315d95a201c02bade8673e122b43c2f9c364660a2"
Sep 10 14:26:53.373676 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:53.373659    2580 scope.go:117] "RemoveContainer" containerID="5de987b85e6b5ca922b4ff4315d95a201c02bade8673e122b43c2f9c364660a2"
Sep 10 14:26:53.373936 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:26:53.373916    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de987b85e6b5ca922b4ff4315d95a201c02bade8673e122b43c2f9c364660a2\": container with ID starting with 5de987b85e6b5ca922b4ff4315d95a201c02bade8673e122b43c2f9c364660a2 not found: ID does not exist" containerID="5de987b85e6b5ca922b4ff4315d95a201c02bade8673e122b43c2f9c364660a2"
Sep 10 14:26:53.374060 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:53.373941    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de987b85e6b5ca922b4ff4315d95a201c02bade8673e122b43c2f9c364660a2"} err="failed to get container status \"5de987b85e6b5ca922b4ff4315d95a201c02bade8673e122b43c2f9c364660a2\": rpc error: code = NotFound desc = could not find container \"5de987b85e6b5ca922b4ff4315d95a201c02bade8673e122b43c2f9c364660a2\": container with ID starting with 5de987b85e6b5ca922b4ff4315d95a201c02bade8673e122b43c2f9c364660a2 not found: ID does not exist"
Sep 10 14:26:53.387189 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:26:53.387162    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2859db2_22aa_4d1a_bc41_db21c90bfe12.slice/crio-f073b5b7d7599f488ff8390dda1406e67365175546c1e42f0a3d26d99931b9c1\": RecentStats: unable to find data in memory cache]"
Sep 10 14:26:53.392911 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:53.392890    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"]
Sep 10 14:26:53.402073 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:53.402045    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-45jpk"]
Sep 10 14:26:54.309873 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:26:54.309836    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2859db2-22aa-4d1a-bc41-db21c90bfe12" path="/var/lib/kubelet/pods/d2859db2-22aa-4d1a-bc41-db21c90bfe12/volumes"
Sep 10 14:27:23.789641 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:23.789602    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mwhsk"]
Sep 10 14:27:23.790105 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:23.789956    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2859db2-22aa-4d1a-bc41-db21c90bfe12" containerName="manager"
Sep 10 14:27:23.790105 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:23.789980    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2859db2-22aa-4d1a-bc41-db21c90bfe12" containerName="manager"
Sep 10 14:27:23.790105 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:23.790059    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2859db2-22aa-4d1a-bc41-db21c90bfe12" containerName="manager"
Sep 10 14:27:23.806537 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:23.806509    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mwhsk"]
Sep 10 14:27:23.806676 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:23.806610    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mwhsk"
Sep 10 14:27:23.809062 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:23.809037    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 14:27:23.823431 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:23.823409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/881e2a2a-c339-40d5-80e8-49abb5a17ef1-config-file\") pod \"limitador-limitador-8588749977-mwhsk\" (UID: \"881e2a2a-c339-40d5-80e8-49abb5a17ef1\") " pod="kuadrant-system/limitador-limitador-8588749977-mwhsk"
Sep 10 14:27:23.823553 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:23.823465    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwcst\" (UniqueName: \"kubernetes.io/projected/881e2a2a-c339-40d5-80e8-49abb5a17ef1-kube-api-access-wwcst\") pod \"limitador-limitador-8588749977-mwhsk\" (UID: \"881e2a2a-c339-40d5-80e8-49abb5a17ef1\") " pod="kuadrant-system/limitador-limitador-8588749977-mwhsk"
Sep 10 14:27:23.883021 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:23.882989    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mwhsk"]
Sep 10 14:27:23.926852 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:23.925127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwcst\" (UniqueName: \"kubernetes.io/projected/881e2a2a-c339-40d5-80e8-49abb5a17ef1-kube-api-access-wwcst\") pod \"limitador-limitador-8588749977-mwhsk\" (UID: \"881e2a2a-c339-40d5-80e8-49abb5a17ef1\") " pod="kuadrant-system/limitador-limitador-8588749977-mwhsk"
Sep 10 14:27:23.926852 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:23.925240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/881e2a2a-c339-40d5-80e8-49abb5a17ef1-config-file\") pod \"limitador-limitador-8588749977-mwhsk\" (UID: \"881e2a2a-c339-40d5-80e8-49abb5a17ef1\") " pod="kuadrant-system/limitador-limitador-8588749977-mwhsk"
Sep 10 14:27:23.926852 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:23.926091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/881e2a2a-c339-40d5-80e8-49abb5a17ef1-config-file\") pod \"limitador-limitador-8588749977-mwhsk\" (UID: \"881e2a2a-c339-40d5-80e8-49abb5a17ef1\") " pod="kuadrant-system/limitador-limitador-8588749977-mwhsk"
Sep 10 14:27:23.933586 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:23.933561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwcst\" (UniqueName: \"kubernetes.io/projected/881e2a2a-c339-40d5-80e8-49abb5a17ef1-kube-api-access-wwcst\") pod \"limitador-limitador-8588749977-mwhsk\" (UID: \"881e2a2a-c339-40d5-80e8-49abb5a17ef1\") " pod="kuadrant-system/limitador-limitador-8588749977-mwhsk"
Sep 10 14:27:24.116703 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:24.116616    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mwhsk"
Sep 10 14:27:24.385402 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:24.385378    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mwhsk"]
Sep 10 14:27:24.387411 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:27:24.387384    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881e2a2a_c339_40d5_80e8_49abb5a17ef1.slice/crio-bf050dcaf1a3586704983e03f6f52673f950da835f92003161b52bc28e7eeb6e WatchSource:0}: Error finding container bf050dcaf1a3586704983e03f6f52673f950da835f92003161b52bc28e7eeb6e: Status 404 returned error can't find the container with id bf050dcaf1a3586704983e03f6f52673f950da835f92003161b52bc28e7eeb6e
Sep 10 14:27:24.525008 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:24.524962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mwhsk" event={"ID":"881e2a2a-c339-40d5-80e8-49abb5a17ef1","Type":"ContainerStarted","Data":"bf050dcaf1a3586704983e03f6f52673f950da835f92003161b52bc28e7eeb6e"}
Sep 10 14:27:24.684650 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:24.684569    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-957nm"]
Sep 10 14:27:24.687503 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:24.687481    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-957nm"
Sep 10 14:27:24.689899 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:24.689879    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ghtw8\""
Sep 10 14:27:24.696198 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:24.696164    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-957nm"]
Sep 10 14:27:24.733463 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:24.733426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffdh\" (UniqueName: \"kubernetes.io/projected/02a5364d-ce8b-484e-b08b-0e141a98fe26-kube-api-access-6ffdh\") pod \"authorino-59c974bd48-957nm\" (UID: \"02a5364d-ce8b-484e-b08b-0e141a98fe26\") " pod="kuadrant-system/authorino-59c974bd48-957nm"
Sep 10 14:27:24.834882 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:24.834830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffdh\" (UniqueName: \"kubernetes.io/projected/02a5364d-ce8b-484e-b08b-0e141a98fe26-kube-api-access-6ffdh\") pod \"authorino-59c974bd48-957nm\" (UID: \"02a5364d-ce8b-484e-b08b-0e141a98fe26\") " pod="kuadrant-system/authorino-59c974bd48-957nm"
Sep 10 14:27:24.843205 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:24.843178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffdh\" (UniqueName: \"kubernetes.io/projected/02a5364d-ce8b-484e-b08b-0e141a98fe26-kube-api-access-6ffdh\") pod \"authorino-59c974bd48-957nm\" (UID: \"02a5364d-ce8b-484e-b08b-0e141a98fe26\") " pod="kuadrant-system/authorino-59c974bd48-957nm"
Sep 10 14:27:24.998334 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:24.998250    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-957nm"
Sep 10 14:27:25.266829 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:25.266741    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-957nm"]
Sep 10 14:27:25.269435 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:27:25.269409    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02a5364d_ce8b_484e_b08b_0e141a98fe26.slice/crio-3c2eb7610c4fba837c745fbef4382df8f4ced40ee9e83b617e7038750681ce78 WatchSource:0}: Error finding container 3c2eb7610c4fba837c745fbef4382df8f4ced40ee9e83b617e7038750681ce78: Status 404 returned error can't find the container with id 3c2eb7610c4fba837c745fbef4382df8f4ced40ee9e83b617e7038750681ce78
Sep 10 14:27:25.532239 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:25.532150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-957nm" event={"ID":"02a5364d-ce8b-484e-b08b-0e141a98fe26","Type":"ContainerStarted","Data":"3c2eb7610c4fba837c745fbef4382df8f4ced40ee9e83b617e7038750681ce78"}
Sep 10 14:27:26.541208 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:26.541167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mwhsk" event={"ID":"881e2a2a-c339-40d5-80e8-49abb5a17ef1","Type":"ContainerStarted","Data":"d0e80ff2ec7067f8dad0ec7959e6e33eb1a4e1f8180fba01125b27fcda629306"}
Sep 10 14:27:26.541671 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:26.541244    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-mwhsk"
Sep 10 14:27:26.559416 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:26.559344    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-mwhsk" podStartSLOduration=1.959796404 podStartE2EDuration="3.559329411s" totalImagesPullingTime="1.599533007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:27:26.557499677 +0000 UTC m=+464.749027876" watchObservedRunningTime="2026-09-10 14:27:26.559329411 +0000 UTC m=+464.750857609"
Sep 10 14:27:27.547960 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:27.547924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-957nm" event={"ID":"02a5364d-ce8b-484e-b08b-0e141a98fe26","Type":"ContainerStarted","Data":"ef9c8187446bdf6a3be07d8692b3034eecf4b6692a5718a99f1c0da07dfad45e"}
Sep 10 14:27:27.563337 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:27.563283    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-957nm" podStartSLOduration=1.704008194 podStartE2EDuration="3.563266966s" totalImagesPullingTime="1.859258772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:27:27.562082136 +0000 UTC m=+465.753610349" watchObservedRunningTime="2026-09-10 14:27:27.563266966 +0000 UTC m=+465.754795163"
Sep 10 14:27:37.549471 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:37.549442    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-mwhsk"
Sep 10 14:27:37.829082 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:37.828995    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mwhsk"]
Sep 10 14:27:37.829237 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:37.829203    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-mwhsk" podUID="881e2a2a-c339-40d5-80e8-49abb5a17ef1" containerName="limitador" containerID="cri-o://d0e80ff2ec7067f8dad0ec7959e6e33eb1a4e1f8180fba01125b27fcda629306" gracePeriod=30
Sep 10 14:27:38.376582 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.376558    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mwhsk"
Sep 10 14:27:38.448325 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.448241    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwcst\" (UniqueName: \"kubernetes.io/projected/881e2a2a-c339-40d5-80e8-49abb5a17ef1-kube-api-access-wwcst\") pod \"881e2a2a-c339-40d5-80e8-49abb5a17ef1\" (UID: \"881e2a2a-c339-40d5-80e8-49abb5a17ef1\") "
Sep 10 14:27:38.448325 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.448286    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/881e2a2a-c339-40d5-80e8-49abb5a17ef1-config-file\") pod \"881e2a2a-c339-40d5-80e8-49abb5a17ef1\" (UID: \"881e2a2a-c339-40d5-80e8-49abb5a17ef1\") "
Sep 10 14:27:38.448671 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.448645    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881e2a2a-c339-40d5-80e8-49abb5a17ef1-config-file" (OuterVolumeSpecName: "config-file") pod "881e2a2a-c339-40d5-80e8-49abb5a17ef1" (UID: "881e2a2a-c339-40d5-80e8-49abb5a17ef1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:27:38.450265 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.450245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881e2a2a-c339-40d5-80e8-49abb5a17ef1-kube-api-access-wwcst" (OuterVolumeSpecName: "kube-api-access-wwcst") pod "881e2a2a-c339-40d5-80e8-49abb5a17ef1" (UID: "881e2a2a-c339-40d5-80e8-49abb5a17ef1"). InnerVolumeSpecName "kube-api-access-wwcst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:27:38.549013 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.548956    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwcst\" (UniqueName: \"kubernetes.io/projected/881e2a2a-c339-40d5-80e8-49abb5a17ef1-kube-api-access-wwcst\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:27:38.549013 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.549010    2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/881e2a2a-c339-40d5-80e8-49abb5a17ef1-config-file\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:27:38.605767 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.605734    2580 generic.go:356] "Generic (PLEG): container finished" podID="881e2a2a-c339-40d5-80e8-49abb5a17ef1" containerID="d0e80ff2ec7067f8dad0ec7959e6e33eb1a4e1f8180fba01125b27fcda629306" exitCode=0
Sep 10 14:27:38.606214 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.605785    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mwhsk"
Sep 10 14:27:38.606214 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.605790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mwhsk" event={"ID":"881e2a2a-c339-40d5-80e8-49abb5a17ef1","Type":"ContainerDied","Data":"d0e80ff2ec7067f8dad0ec7959e6e33eb1a4e1f8180fba01125b27fcda629306"}
Sep 10 14:27:38.606214 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.605910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mwhsk" event={"ID":"881e2a2a-c339-40d5-80e8-49abb5a17ef1","Type":"ContainerDied","Data":"bf050dcaf1a3586704983e03f6f52673f950da835f92003161b52bc28e7eeb6e"}
Sep 10 14:27:38.606214 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.605934    2580 scope.go:117] "RemoveContainer" containerID="d0e80ff2ec7067f8dad0ec7959e6e33eb1a4e1f8180fba01125b27fcda629306"
Sep 10 14:27:38.614744 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.614729    2580 scope.go:117] "RemoveContainer" containerID="d0e80ff2ec7067f8dad0ec7959e6e33eb1a4e1f8180fba01125b27fcda629306"
Sep 10 14:27:38.615032 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:27:38.615008    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e80ff2ec7067f8dad0ec7959e6e33eb1a4e1f8180fba01125b27fcda629306\": container with ID starting with d0e80ff2ec7067f8dad0ec7959e6e33eb1a4e1f8180fba01125b27fcda629306 not found: ID does not exist" containerID="d0e80ff2ec7067f8dad0ec7959e6e33eb1a4e1f8180fba01125b27fcda629306"
Sep 10 14:27:38.615078 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.615038    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e80ff2ec7067f8dad0ec7959e6e33eb1a4e1f8180fba01125b27fcda629306"} err="failed to get container status \"d0e80ff2ec7067f8dad0ec7959e6e33eb1a4e1f8180fba01125b27fcda629306\": rpc error: code = NotFound desc = could not find container \"d0e80ff2ec7067f8dad0ec7959e6e33eb1a4e1f8180fba01125b27fcda629306\": container with ID starting with d0e80ff2ec7067f8dad0ec7959e6e33eb1a4e1f8180fba01125b27fcda629306 not found: ID does not exist"
Sep 10 14:27:38.626621 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.626598    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mwhsk"]
Sep 10 14:27:38.630012 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:38.629991    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mwhsk"]
Sep 10 14:27:40.312128 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:40.309896    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="881e2a2a-c339-40d5-80e8-49abb5a17ef1" path="/var/lib/kubelet/pods/881e2a2a-c339-40d5-80e8-49abb5a17ef1/volumes"
Sep 10 14:27:52.855390 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:52.855354    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-p6k86"]
Sep 10 14:27:52.855859 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:52.855702    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="881e2a2a-c339-40d5-80e8-49abb5a17ef1" containerName="limitador"
Sep 10 14:27:52.855859 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:52.855716    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="881e2a2a-c339-40d5-80e8-49abb5a17ef1" containerName="limitador"
Sep 10 14:27:52.855859 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:52.855777    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="881e2a2a-c339-40d5-80e8-49abb5a17ef1" containerName="limitador"
Sep 10 14:27:52.858748 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:52.858732    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-p6k86"
Sep 10 14:27:52.863723 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:52.863694    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-p6k86"]
Sep 10 14:27:52.909463 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:52.909428    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-p6k86"]
Sep 10 14:27:52.909727 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:27:52.909704    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-g255l], unattached volumes=[], failed to process volumes=[kube-api-access-g255l]: context canceled" pod="kuadrant-system/authorino-557f876f69-p6k86" podUID="7a672f63-837e-4fd1-ae82-5e06daaf50b6"
Sep 10 14:27:52.975388 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:52.975354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g255l\" (UniqueName: \"kubernetes.io/projected/7a672f63-837e-4fd1-ae82-5e06daaf50b6-kube-api-access-g255l\") pod \"authorino-557f876f69-p6k86\" (UID: \"7a672f63-837e-4fd1-ae82-5e06daaf50b6\") " pod="kuadrant-system/authorino-557f876f69-p6k86"
Sep 10 14:27:53.076704 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.076665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g255l\" (UniqueName: \"kubernetes.io/projected/7a672f63-837e-4fd1-ae82-5e06daaf50b6-kube-api-access-g255l\") pod \"authorino-557f876f69-p6k86\" (UID: \"7a672f63-837e-4fd1-ae82-5e06daaf50b6\") " pod="kuadrant-system/authorino-557f876f69-p6k86"
Sep 10 14:27:53.084624 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.084592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g255l\" (UniqueName: \"kubernetes.io/projected/7a672f63-837e-4fd1-ae82-5e06daaf50b6-kube-api-access-g255l\") pod \"authorino-557f876f69-p6k86\" (UID: \"7a672f63-837e-4fd1-ae82-5e06daaf50b6\") " pod="kuadrant-system/authorino-557f876f69-p6k86"
Sep 10 14:27:53.127454 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.124627    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cfffdfc98-p6wwb"]
Sep 10 14:27:53.128587 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.128557    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cfffdfc98-p6wwb"
Sep 10 14:27:53.131234 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.131209    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 14:27:53.132858 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.132835    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cfffdfc98-p6wwb"]
Sep 10 14:27:53.278058 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.278019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jjct\" (UniqueName: \"kubernetes.io/projected/4f27ed99-a646-413e-b610-c7e5265f05ad-kube-api-access-2jjct\") pod \"authorino-6cfffdfc98-p6wwb\" (UID: \"4f27ed99-a646-413e-b610-c7e5265f05ad\") " pod="kuadrant-system/authorino-6cfffdfc98-p6wwb"
Sep 10 14:27:53.278233 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.278089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4f27ed99-a646-413e-b610-c7e5265f05ad-tls-cert\") pod \"authorino-6cfffdfc98-p6wwb\" (UID: \"4f27ed99-a646-413e-b610-c7e5265f05ad\") " pod="kuadrant-system/authorino-6cfffdfc98-p6wwb"
Sep 10 14:27:53.379152 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.379064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4f27ed99-a646-413e-b610-c7e5265f05ad-tls-cert\") pod \"authorino-6cfffdfc98-p6wwb\" (UID: \"4f27ed99-a646-413e-b610-c7e5265f05ad\") " pod="kuadrant-system/authorino-6cfffdfc98-p6wwb"
Sep 10 14:27:53.379152 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.379131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjct\" (UniqueName: \"kubernetes.io/projected/4f27ed99-a646-413e-b610-c7e5265f05ad-kube-api-access-2jjct\") pod \"authorino-6cfffdfc98-p6wwb\" (UID: \"4f27ed99-a646-413e-b610-c7e5265f05ad\") " pod="kuadrant-system/authorino-6cfffdfc98-p6wwb"
Sep 10 14:27:53.381488 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.381462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4f27ed99-a646-413e-b610-c7e5265f05ad-tls-cert\") pod \"authorino-6cfffdfc98-p6wwb\" (UID: \"4f27ed99-a646-413e-b610-c7e5265f05ad\") " pod="kuadrant-system/authorino-6cfffdfc98-p6wwb"
Sep 10 14:27:53.387036 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.387007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjct\" (UniqueName: \"kubernetes.io/projected/4f27ed99-a646-413e-b610-c7e5265f05ad-kube-api-access-2jjct\") pod \"authorino-6cfffdfc98-p6wwb\" (UID: \"4f27ed99-a646-413e-b610-c7e5265f05ad\") " pod="kuadrant-system/authorino-6cfffdfc98-p6wwb"
Sep 10 14:27:53.438872 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.438823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cfffdfc98-p6wwb"
Sep 10 14:27:53.682128 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.682097    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-p6k86"
Sep 10 14:27:53.688402 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.688378    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-p6k86"
Sep 10 14:27:53.710598 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.710574    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cfffdfc98-p6wwb"]
Sep 10 14:27:53.712362 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:27:53.712338    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f27ed99_a646_413e_b610_c7e5265f05ad.slice/crio-d5f7af95e946f621b62d7f7d84d5b9602c5bb1f272c955974bf481a8bbd04771 WatchSource:0}: Error finding container d5f7af95e946f621b62d7f7d84d5b9602c5bb1f272c955974bf481a8bbd04771: Status 404 returned error can't find the container with id d5f7af95e946f621b62d7f7d84d5b9602c5bb1f272c955974bf481a8bbd04771
Sep 10 14:27:53.781759 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.781722    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g255l\" (UniqueName: \"kubernetes.io/projected/7a672f63-837e-4fd1-ae82-5e06daaf50b6-kube-api-access-g255l\") pod \"7a672f63-837e-4fd1-ae82-5e06daaf50b6\" (UID: \"7a672f63-837e-4fd1-ae82-5e06daaf50b6\") "
Sep 10 14:27:53.783955 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.783926    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a672f63-837e-4fd1-ae82-5e06daaf50b6-kube-api-access-g255l" (OuterVolumeSpecName: "kube-api-access-g255l") pod "7a672f63-837e-4fd1-ae82-5e06daaf50b6" (UID: "7a672f63-837e-4fd1-ae82-5e06daaf50b6"). InnerVolumeSpecName "kube-api-access-g255l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:27:53.883198 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:53.883159    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g255l\" (UniqueName: \"kubernetes.io/projected/7a672f63-837e-4fd1-ae82-5e06daaf50b6-kube-api-access-g255l\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:27:54.688266 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:54.688235    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-p6k86"
Sep 10 14:27:54.688266 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:54.688247    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cfffdfc98-p6wwb" event={"ID":"4f27ed99-a646-413e-b610-c7e5265f05ad","Type":"ContainerStarted","Data":"d5f7af95e946f621b62d7f7d84d5b9602c5bb1f272c955974bf481a8bbd04771"}
Sep 10 14:27:54.715116 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:54.715089    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-p6k86"]
Sep 10 14:27:54.720103 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:54.720075    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-p6k86"]
Sep 10 14:27:55.695015 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:55.694958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cfffdfc98-p6wwb" event={"ID":"4f27ed99-a646-413e-b610-c7e5265f05ad","Type":"ContainerStarted","Data":"ae11bd6d164875395856c72f639ce9e0a5b19f727fad78a4f8f552bf9526da92"}
Sep 10 14:27:55.711522 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:55.711468    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6cfffdfc98-p6wwb" podStartSLOduration=1.7148038620000001 podStartE2EDuration="2.711452358s" totalImagesPullingTime="996.648496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:27:55.709737024 +0000 UTC m=+493.901265214" watchObservedRunningTime="2026-09-10 14:27:55.711452358 +0000 UTC m=+493.902980555"
Sep 10 14:27:55.736051 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:55.736017    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-957nm"]
Sep 10 14:27:55.736246 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:55.736218    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-957nm" podUID="02a5364d-ce8b-484e-b08b-0e141a98fe26" containerName="authorino" containerID="cri-o://ef9c8187446bdf6a3be07d8692b3034eecf4b6692a5718a99f1c0da07dfad45e" gracePeriod=30
Sep 10 14:27:55.974063 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:55.974040    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-957nm"
Sep 10 14:27:56.102184 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:56.102150    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ffdh\" (UniqueName: \"kubernetes.io/projected/02a5364d-ce8b-484e-b08b-0e141a98fe26-kube-api-access-6ffdh\") pod \"02a5364d-ce8b-484e-b08b-0e141a98fe26\" (UID: \"02a5364d-ce8b-484e-b08b-0e141a98fe26\") "
Sep 10 14:27:56.104234 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:56.104202    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a5364d-ce8b-484e-b08b-0e141a98fe26-kube-api-access-6ffdh" (OuterVolumeSpecName: "kube-api-access-6ffdh") pod "02a5364d-ce8b-484e-b08b-0e141a98fe26" (UID: "02a5364d-ce8b-484e-b08b-0e141a98fe26"). InnerVolumeSpecName "kube-api-access-6ffdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:27:56.203757 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:56.203718    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ffdh\" (UniqueName: \"kubernetes.io/projected/02a5364d-ce8b-484e-b08b-0e141a98fe26-kube-api-access-6ffdh\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:27:56.310505 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:56.310428    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a672f63-837e-4fd1-ae82-5e06daaf50b6" path="/var/lib/kubelet/pods/7a672f63-837e-4fd1-ae82-5e06daaf50b6/volumes"
Sep 10 14:27:56.700739 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:56.700701    2580 generic.go:356] "Generic (PLEG): container finished" podID="02a5364d-ce8b-484e-b08b-0e141a98fe26" containerID="ef9c8187446bdf6a3be07d8692b3034eecf4b6692a5718a99f1c0da07dfad45e" exitCode=0
Sep 10 14:27:56.701242 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:56.700765    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-957nm"
Sep 10 14:27:56.701242 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:56.700788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-957nm" event={"ID":"02a5364d-ce8b-484e-b08b-0e141a98fe26","Type":"ContainerDied","Data":"ef9c8187446bdf6a3be07d8692b3034eecf4b6692a5718a99f1c0da07dfad45e"}
Sep 10 14:27:56.701242 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:56.700830    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-957nm" event={"ID":"02a5364d-ce8b-484e-b08b-0e141a98fe26","Type":"ContainerDied","Data":"3c2eb7610c4fba837c745fbef4382df8f4ced40ee9e83b617e7038750681ce78"}
Sep 10 14:27:56.701242 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:56.700847    2580 scope.go:117] "RemoveContainer" containerID="ef9c8187446bdf6a3be07d8692b3034eecf4b6692a5718a99f1c0da07dfad45e"
Sep 10 14:27:56.709630 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:56.709609    2580 scope.go:117] "RemoveContainer" containerID="ef9c8187446bdf6a3be07d8692b3034eecf4b6692a5718a99f1c0da07dfad45e"
Sep 10 14:27:56.709936 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:27:56.709915    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9c8187446bdf6a3be07d8692b3034eecf4b6692a5718a99f1c0da07dfad45e\": container with ID starting with ef9c8187446bdf6a3be07d8692b3034eecf4b6692a5718a99f1c0da07dfad45e not found: ID does not exist" containerID="ef9c8187446bdf6a3be07d8692b3034eecf4b6692a5718a99f1c0da07dfad45e"
Sep 10 14:27:56.710002 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:56.709948    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9c8187446bdf6a3be07d8692b3034eecf4b6692a5718a99f1c0da07dfad45e"} err="failed to get container status \"ef9c8187446bdf6a3be07d8692b3034eecf4b6692a5718a99f1c0da07dfad45e\": rpc error: code = NotFound desc = could not find container \"ef9c8187446bdf6a3be07d8692b3034eecf4b6692a5718a99f1c0da07dfad45e\": container with ID starting with ef9c8187446bdf6a3be07d8692b3034eecf4b6692a5718a99f1c0da07dfad45e not found: ID does not exist"
Sep 10 14:27:56.717190 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:56.717165    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-957nm"]
Sep 10 14:27:56.720880 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:56.720855    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-957nm"]
Sep 10 14:27:58.310341 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:27:58.310307    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02a5364d-ce8b-484e-b08b-0e141a98fe26" path="/var/lib/kubelet/pods/02a5364d-ce8b-484e-b08b-0e141a98fe26/volumes"
Sep 10 14:28:00.483901 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.483858    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57f767fb74-778v6"]
Sep 10 14:28:00.484312 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.484224    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a5364d-ce8b-484e-b08b-0e141a98fe26" containerName="authorino"
Sep 10 14:28:00.484312 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.484235    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a5364d-ce8b-484e-b08b-0e141a98fe26" containerName="authorino"
Sep 10 14:28:00.484394 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.484313    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="02a5364d-ce8b-484e-b08b-0e141a98fe26" containerName="authorino"
Sep 10 14:28:00.488371 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.488350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:00.491024 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.490996    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 14:28:00.491178 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.491027    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 14:28:00.491178 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.491087    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 14:28:00.491178 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.491144    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tbsjp\""
Sep 10 14:28:00.500133 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.500107    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57f767fb74-778v6"]
Sep 10 14:28:00.645024 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.644985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-webhook-cert\") pod \"maas-controller-57f767fb74-778v6\" (UID: \"839cbf1c-7128-4355-9f0d-f9034f3e6646\") " pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:00.645229 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.645120    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-metrics-certs\") pod \"maas-controller-57f767fb74-778v6\" (UID: \"839cbf1c-7128-4355-9f0d-f9034f3e6646\") " pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:00.645229 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.645151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8jb4\" (UniqueName: \"kubernetes.io/projected/839cbf1c-7128-4355-9f0d-f9034f3e6646-kube-api-access-b8jb4\") pod \"maas-controller-57f767fb74-778v6\" (UID: \"839cbf1c-7128-4355-9f0d-f9034f3e6646\") " pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:00.647924 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.647895    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6857cb7d9d-gst9r"]
Sep 10 14:28:00.651179 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.651163    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:28:00.662017 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.661991    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6857cb7d9d-gst9r"]
Sep 10 14:28:00.746362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.746275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-metrics-certs\") pod \"maas-controller-57f767fb74-778v6\" (UID: \"839cbf1c-7128-4355-9f0d-f9034f3e6646\") " pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:00.746362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.746310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8jb4\" (UniqueName: \"kubernetes.io/projected/839cbf1c-7128-4355-9f0d-f9034f3e6646-kube-api-access-b8jb4\") pod \"maas-controller-57f767fb74-778v6\" (UID: \"839cbf1c-7128-4355-9f0d-f9034f3e6646\") " pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:00.746362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.746345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-metrics-certs\") pod \"maas-controller-6857cb7d9d-gst9r\" (UID: \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\") " pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:28:00.746620 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.746373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-webhook-cert\") pod \"maas-controller-57f767fb74-778v6\" (UID: \"839cbf1c-7128-4355-9f0d-f9034f3e6646\") " pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:00.746620 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.746389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgzz8\" (UniqueName: \"kubernetes.io/projected/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-kube-api-access-tgzz8\") pod \"maas-controller-6857cb7d9d-gst9r\" (UID: \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\") " pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:28:00.746620 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.746428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-webhook-cert\") pod \"maas-controller-6857cb7d9d-gst9r\" (UID: \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\") " pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:28:00.746620 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:28:00.746435    2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 10 14:28:00.746620 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:28:00.746530    2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 10 14:28:00.746620 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:28:00.746549    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-metrics-certs podName:839cbf1c-7128-4355-9f0d-f9034f3e6646 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:01.246527873 +0000 UTC m=+499.438056063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-metrics-certs") pod "maas-controller-57f767fb74-778v6" (UID: "839cbf1c-7128-4355-9f0d-f9034f3e6646") : secret "maas-controller-metrics-tls" not found
Sep 10 14:28:00.746620 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:28:00.746576    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-webhook-cert podName:839cbf1c-7128-4355-9f0d-f9034f3e6646 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:01.246567528 +0000 UTC m=+499.438095718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-webhook-cert") pod "maas-controller-57f767fb74-778v6" (UID: "839cbf1c-7128-4355-9f0d-f9034f3e6646") : secret "maas-controller-webhook-cert" not found
Sep 10 14:28:00.757615 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.757583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8jb4\" (UniqueName: \"kubernetes.io/projected/839cbf1c-7128-4355-9f0d-f9034f3e6646-kube-api-access-b8jb4\") pod \"maas-controller-57f767fb74-778v6\" (UID: \"839cbf1c-7128-4355-9f0d-f9034f3e6646\") " pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:00.847749 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.847715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-metrics-certs\") pod \"maas-controller-6857cb7d9d-gst9r\" (UID: \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\") " pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:28:00.847951 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.847760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgzz8\" (UniqueName: \"kubernetes.io/projected/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-kube-api-access-tgzz8\") pod \"maas-controller-6857cb7d9d-gst9r\" (UID: \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\") " pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:28:00.847951 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.847796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-webhook-cert\") pod \"maas-controller-6857cb7d9d-gst9r\" (UID: \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\") " pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:28:00.847951 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:28:00.847924    2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 10 14:28:00.848128 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:28:00.848013    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-webhook-cert podName:7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d nodeName:}" failed. No retries permitted until 2026-09-10 14:28:01.347967594 +0000 UTC m=+499.539495788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-webhook-cert") pod "maas-controller-6857cb7d9d-gst9r" (UID: "7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d") : secret "maas-controller-webhook-cert" not found
Sep 10 14:28:00.850342 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.850317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-metrics-certs\") pod \"maas-controller-6857cb7d9d-gst9r\" (UID: \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\") " pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:28:00.855609 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:00.855583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgzz8\" (UniqueName: \"kubernetes.io/projected/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-kube-api-access-tgzz8\") pod \"maas-controller-6857cb7d9d-gst9r\" (UID: \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\") " pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:28:01.252226 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:01.252175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-metrics-certs\") pod \"maas-controller-57f767fb74-778v6\" (UID: \"839cbf1c-7128-4355-9f0d-f9034f3e6646\") " pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:01.252226 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:01.252244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-webhook-cert\") pod \"maas-controller-57f767fb74-778v6\" (UID: \"839cbf1c-7128-4355-9f0d-f9034f3e6646\") " pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:01.254676 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:01.254650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-metrics-certs\") pod \"maas-controller-57f767fb74-778v6\" (UID: \"839cbf1c-7128-4355-9f0d-f9034f3e6646\") " pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:01.254781 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:01.254717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-webhook-cert\") pod \"maas-controller-57f767fb74-778v6\" (UID: \"839cbf1c-7128-4355-9f0d-f9034f3e6646\") " pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:01.353206 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:01.353154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-webhook-cert\") pod \"maas-controller-6857cb7d9d-gst9r\" (UID: \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\") " pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:28:01.355522 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:01.355501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-webhook-cert\") pod \"maas-controller-6857cb7d9d-gst9r\" (UID: \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\") " pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:28:01.400621 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:01.400586    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:01.562274 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:01.562175    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:28:01.676469 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:28:01.676432    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839cbf1c_7128_4355_9f0d_f9034f3e6646.slice/crio-117247ebcc5e86d8f88e0ecaefaef4dc3093c7a1cc8f448b3a7acfe45597a6ad WatchSource:0}: Error finding container 117247ebcc5e86d8f88e0ecaefaef4dc3093c7a1cc8f448b3a7acfe45597a6ad: Status 404 returned error can't find the container with id 117247ebcc5e86d8f88e0ecaefaef4dc3093c7a1cc8f448b3a7acfe45597a6ad
Sep 10 14:28:01.676669 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:01.676632    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57f767fb74-778v6"]
Sep 10 14:28:01.728540 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:01.728502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57f767fb74-778v6" event={"ID":"839cbf1c-7128-4355-9f0d-f9034f3e6646","Type":"ContainerStarted","Data":"117247ebcc5e86d8f88e0ecaefaef4dc3093c7a1cc8f448b3a7acfe45597a6ad"}
Sep 10 14:28:01.834217 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:01.834191    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6857cb7d9d-gst9r"]
Sep 10 14:28:01.835955 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:28:01.835926    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af9765c_c2d0_4c1b_86a4_9aeb6cd62e5d.slice/crio-fc05810e88f0bab0c189a4fa923cf2f40945d9421392a43222f8bb2e548fd0a4 WatchSource:0}: Error finding container fc05810e88f0bab0c189a4fa923cf2f40945d9421392a43222f8bb2e548fd0a4: Status 404 returned error can't find the container with id fc05810e88f0bab0c189a4fa923cf2f40945d9421392a43222f8bb2e548fd0a4
Sep 10 14:28:02.744135 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:02.744033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6857cb7d9d-gst9r" event={"ID":"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d","Type":"ContainerStarted","Data":"fc05810e88f0bab0c189a4fa923cf2f40945d9421392a43222f8bb2e548fd0a4"}
Sep 10 14:28:04.757640 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:04.757593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57f767fb74-778v6" event={"ID":"839cbf1c-7128-4355-9f0d-f9034f3e6646","Type":"ContainerStarted","Data":"d17cbb8c36987c0a01db75031fb689aa849ff30b0b19fbc408c68044964dab65"}
Sep 10 14:28:04.758156 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:04.757782    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:04.759252 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:04.759228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6857cb7d9d-gst9r" event={"ID":"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d","Type":"ContainerStarted","Data":"fff57f019b5dd1583a3d32a3e62ecc3499233f39246ff6c2fbf0bd85004a9498"}
Sep 10 14:28:04.759390 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:04.759275    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:28:04.776403 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:04.776348    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57f767fb74-778v6" podStartSLOduration=2.074488509 podStartE2EDuration="4.776332595s" totalImagesPullingTime="2.701844086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:28:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:04.775135979 +0000 UTC m=+502.966664177" watchObservedRunningTime="2026-09-10 14:28:04.776332595 +0000 UTC m=+502.967860793"
Sep 10 14:28:04.798007 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:04.795932    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6857cb7d9d-gst9r" podStartSLOduration=2.259373375 podStartE2EDuration="4.795910752s" totalImagesPullingTime="2.536537377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:28:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:04.792534087 +0000 UTC m=+502.984062284" watchObservedRunningTime="2026-09-10 14:28:04.795910752 +0000 UTC m=+502.987438950"
Sep 10 14:28:15.767486 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:15.767455    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:28:15.767942 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:15.767717    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:15.832113 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:15.832079    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57f767fb74-778v6"]
Sep 10 14:28:15.832324 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:15.832300    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57f767fb74-778v6" podUID="839cbf1c-7128-4355-9f0d-f9034f3e6646" containerName="manager" containerID="cri-o://d17cbb8c36987c0a01db75031fb689aa849ff30b0b19fbc408c68044964dab65" gracePeriod=30
Sep 10 14:28:16.075195 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.075170    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:16.178828 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.178787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-webhook-cert\") pod \"839cbf1c-7128-4355-9f0d-f9034f3e6646\" (UID: \"839cbf1c-7128-4355-9f0d-f9034f3e6646\") "
Sep 10 14:28:16.178828 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.178828    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-metrics-certs\") pod \"839cbf1c-7128-4355-9f0d-f9034f3e6646\" (UID: \"839cbf1c-7128-4355-9f0d-f9034f3e6646\") "
Sep 10 14:28:16.179131 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.178884    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8jb4\" (UniqueName: \"kubernetes.io/projected/839cbf1c-7128-4355-9f0d-f9034f3e6646-kube-api-access-b8jb4\") pod \"839cbf1c-7128-4355-9f0d-f9034f3e6646\" (UID: \"839cbf1c-7128-4355-9f0d-f9034f3e6646\") "
Sep 10 14:28:16.181131 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.181090    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "839cbf1c-7128-4355-9f0d-f9034f3e6646" (UID: "839cbf1c-7128-4355-9f0d-f9034f3e6646"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:28:16.181131 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.181126    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "839cbf1c-7128-4355-9f0d-f9034f3e6646" (UID: "839cbf1c-7128-4355-9f0d-f9034f3e6646"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:28:16.181319 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.181154    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839cbf1c-7128-4355-9f0d-f9034f3e6646-kube-api-access-b8jb4" (OuterVolumeSpecName: "kube-api-access-b8jb4") pod "839cbf1c-7128-4355-9f0d-f9034f3e6646" (UID: "839cbf1c-7128-4355-9f0d-f9034f3e6646"). InnerVolumeSpecName "kube-api-access-b8jb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:28:16.280242 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.280205    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-webhook-cert\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:28:16.280242 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.280233    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cbf1c-7128-4355-9f0d-f9034f3e6646-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:28:16.280242 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.280243    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8jb4\" (UniqueName: \"kubernetes.io/projected/839cbf1c-7128-4355-9f0d-f9034f3e6646-kube-api-access-b8jb4\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:28:16.824792 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.824759    2580 generic.go:356] "Generic (PLEG): container finished" podID="839cbf1c-7128-4355-9f0d-f9034f3e6646" containerID="d17cbb8c36987c0a01db75031fb689aa849ff30b0b19fbc408c68044964dab65" exitCode=0
Sep 10 14:28:16.825229 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.824805    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57f767fb74-778v6"
Sep 10 14:28:16.825229 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.824826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57f767fb74-778v6" event={"ID":"839cbf1c-7128-4355-9f0d-f9034f3e6646","Type":"ContainerDied","Data":"d17cbb8c36987c0a01db75031fb689aa849ff30b0b19fbc408c68044964dab65"}
Sep 10 14:28:16.825229 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.824864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57f767fb74-778v6" event={"ID":"839cbf1c-7128-4355-9f0d-f9034f3e6646","Type":"ContainerDied","Data":"117247ebcc5e86d8f88e0ecaefaef4dc3093c7a1cc8f448b3a7acfe45597a6ad"}
Sep 10 14:28:16.825229 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.824886    2580 scope.go:117] "RemoveContainer" containerID="d17cbb8c36987c0a01db75031fb689aa849ff30b0b19fbc408c68044964dab65"
Sep 10 14:28:16.834007 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.833764    2580 scope.go:117] "RemoveContainer" containerID="d17cbb8c36987c0a01db75031fb689aa849ff30b0b19fbc408c68044964dab65"
Sep 10 14:28:16.834076 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:28:16.834041    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17cbb8c36987c0a01db75031fb689aa849ff30b0b19fbc408c68044964dab65\": container with ID starting with d17cbb8c36987c0a01db75031fb689aa849ff30b0b19fbc408c68044964dab65 not found: ID does not exist" containerID="d17cbb8c36987c0a01db75031fb689aa849ff30b0b19fbc408c68044964dab65"
Sep 10 14:28:16.834076 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.834062    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17cbb8c36987c0a01db75031fb689aa849ff30b0b19fbc408c68044964dab65"} err="failed to get container status \"d17cbb8c36987c0a01db75031fb689aa849ff30b0b19fbc408c68044964dab65\": rpc error: code = NotFound desc = could not find container \"d17cbb8c36987c0a01db75031fb689aa849ff30b0b19fbc408c68044964dab65\": container with ID starting with d17cbb8c36987c0a01db75031fb689aa849ff30b0b19fbc408c68044964dab65 not found: ID does not exist"
Sep 10 14:28:16.841188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.841160    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57f767fb74-778v6"]
Sep 10 14:28:16.845247 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:16.845227    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57f767fb74-778v6"]
Sep 10 14:28:18.310245 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:18.310208    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="839cbf1c-7128-4355-9f0d-f9034f3e6646" path="/var/lib/kubelet/pods/839cbf1c-7128-4355-9f0d-f9034f3e6646/volumes"
Sep 10 14:28:38.023014 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.022920    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"]
Sep 10 14:28:38.023463 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.023287    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839cbf1c-7128-4355-9f0d-f9034f3e6646" containerName="manager"
Sep 10 14:28:38.023463 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.023298    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="839cbf1c-7128-4355-9f0d-f9034f3e6646" containerName="manager"
Sep 10 14:28:38.023463 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.023378    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="839cbf1c-7128-4355-9f0d-f9034f3e6646" containerName="manager"
Sep 10 14:28:38.025377 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.025361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:38.027991 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.027948    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 14:28:38.027991 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.027966    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vgr2z\""
Sep 10 14:28:38.029098 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.029080    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 14:28:38.029163 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.029080    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 14:28:38.031073 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.031047    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 14:28:38.037744 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.037721    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"]
Sep 10 14:28:38.172230 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.172194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a75836-b470-4200-96d0-717480b4ddbc-openshift-service-ca\") pod \"maas-api-f96f74b55-zrjbk\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") " pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:38.172230 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.172228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a75836-b470-4200-96d0-717480b4ddbc-metrics-certs\") pod \"maas-api-f96f74b55-zrjbk\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") " pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:38.172432 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.172276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz75w\" (UniqueName: \"kubernetes.io/projected/98a75836-b470-4200-96d0-717480b4ddbc-kube-api-access-hz75w\") pod \"maas-api-f96f74b55-zrjbk\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") " pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:38.172432 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.172306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a75836-b470-4200-96d0-717480b4ddbc-maas-api-tls\") pod \"maas-api-f96f74b55-zrjbk\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") " pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:38.273291 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.273210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a75836-b470-4200-96d0-717480b4ddbc-openshift-service-ca\") pod \"maas-api-f96f74b55-zrjbk\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") " pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:38.273291 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.273246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a75836-b470-4200-96d0-717480b4ddbc-metrics-certs\") pod \"maas-api-f96f74b55-zrjbk\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") " pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:38.273465 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.273291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz75w\" (UniqueName: \"kubernetes.io/projected/98a75836-b470-4200-96d0-717480b4ddbc-kube-api-access-hz75w\") pod \"maas-api-f96f74b55-zrjbk\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") " pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:38.273465 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.273324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a75836-b470-4200-96d0-717480b4ddbc-maas-api-tls\") pod \"maas-api-f96f74b55-zrjbk\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") " pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:38.273988 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.273951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a75836-b470-4200-96d0-717480b4ddbc-openshift-service-ca\") pod \"maas-api-f96f74b55-zrjbk\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") " pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:38.275780 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.275754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a75836-b470-4200-96d0-717480b4ddbc-metrics-certs\") pod \"maas-api-f96f74b55-zrjbk\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") " pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:38.275881 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.275805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a75836-b470-4200-96d0-717480b4ddbc-maas-api-tls\") pod \"maas-api-f96f74b55-zrjbk\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") " pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:38.303026 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.302996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz75w\" (UniqueName: \"kubernetes.io/projected/98a75836-b470-4200-96d0-717480b4ddbc-kube-api-access-hz75w\") pod \"maas-api-f96f74b55-zrjbk\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") " pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:38.336755 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.336721    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:38.615788 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.615758    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"]
Sep 10 14:28:38.618050 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:28:38.618016    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a75836_b470_4200_96d0_717480b4ddbc.slice/crio-5e2626341c070105ea1ab447ae395bbb0c41c181b24ae48f1d1f355d268e6770 WatchSource:0}: Error finding container 5e2626341c070105ea1ab447ae395bbb0c41c181b24ae48f1d1f355d268e6770: Status 404 returned error can't find the container with id 5e2626341c070105ea1ab447ae395bbb0c41c181b24ae48f1d1f355d268e6770
Sep 10 14:28:38.954825 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:38.954787    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk" event={"ID":"98a75836-b470-4200-96d0-717480b4ddbc","Type":"ContainerStarted","Data":"5e2626341c070105ea1ab447ae395bbb0c41c181b24ae48f1d1f355d268e6770"}
Sep 10 14:28:40.966623 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:40.966577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk" event={"ID":"98a75836-b470-4200-96d0-717480b4ddbc","Type":"ContainerStarted","Data":"f5ba541811133ac26ee544c553a33e956bb7c5cf0ac9d96af1e7232f3fe34edf"}
Sep 10 14:28:40.967025 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:40.966656    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:40.987281 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:40.987222    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk" podStartSLOduration=1.026661943 podStartE2EDuration="2.987206148s" totalImagesPullingTime="1.960544205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:40.985300992 +0000 UTC m=+539.176829182" watchObservedRunningTime="2026-09-10 14:28:40.987206148 +0000 UTC m=+539.178734384"
Sep 10 14:28:46.977296 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:46.977267    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:28:57.023026 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.022997    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"]
Sep 10 14:28:57.026144 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.026128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.029669 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.029644    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 10 14:28:57.029787 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.029667    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2pwwc\""
Sep 10 14:28:57.029787 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.029685    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 14:28:57.029787 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.029644    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 14:28:57.035463 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.035439    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"]
Sep 10 14:28:57.038165 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.038139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36135db2-697e-40d0-9824-57128be4bf3a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.038283 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.038190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36135db2-697e-40d0-9824-57128be4bf3a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.038283 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.038226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtdh7\" (UniqueName: \"kubernetes.io/projected/36135db2-697e-40d0-9824-57128be4bf3a-kube-api-access-wtdh7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.038283 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.038278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36135db2-697e-40d0-9824-57128be4bf3a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.038415 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.038312    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36135db2-697e-40d0-9824-57128be4bf3a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.038415 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.038328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36135db2-697e-40d0-9824-57128be4bf3a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.138787 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.138745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36135db2-697e-40d0-9824-57128be4bf3a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.139003 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.138796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36135db2-697e-40d0-9824-57128be4bf3a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.139003 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.138830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtdh7\" (UniqueName: \"kubernetes.io/projected/36135db2-697e-40d0-9824-57128be4bf3a-kube-api-access-wtdh7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.139003 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.138861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36135db2-697e-40d0-9824-57128be4bf3a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.139003 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.138887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36135db2-697e-40d0-9824-57128be4bf3a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.139003 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.138915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36135db2-697e-40d0-9824-57128be4bf3a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.139244 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.139181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36135db2-697e-40d0-9824-57128be4bf3a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.139326 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.139303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36135db2-697e-40d0-9824-57128be4bf3a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.139386 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.139372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36135db2-697e-40d0-9824-57128be4bf3a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.141085 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.141065    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36135db2-697e-40d0-9824-57128be4bf3a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.141384 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.141370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36135db2-697e-40d0-9824-57128be4bf3a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.146484 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.146464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtdh7\" (UniqueName: \"kubernetes.io/projected/36135db2-697e-40d0-9824-57128be4bf3a-kube-api-access-wtdh7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-css28\" (UID: \"36135db2-697e-40d0-9824-57128be4bf3a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.337225 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.337138    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:28:57.608997 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:57.608940    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"]
Sep 10 14:28:57.610946 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:28:57.610922    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36135db2_697e_40d0_9824_57128be4bf3a.slice/crio-de4ee0ed4050859a9429933e1f6fda56915ccb206eec91d41667e3706590b9ef WatchSource:0}: Error finding container de4ee0ed4050859a9429933e1f6fda56915ccb206eec91d41667e3706590b9ef: Status 404 returned error can't find the container with id de4ee0ed4050859a9429933e1f6fda56915ccb206eec91d41667e3706590b9ef
Sep 10 14:28:58.056256 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:58.056176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28" event={"ID":"36135db2-697e-40d0-9824-57128be4bf3a","Type":"ContainerStarted","Data":"de4ee0ed4050859a9429933e1f6fda56915ccb206eec91d41667e3706590b9ef"}
Sep 10 14:28:59.063589 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:59.063491    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28" event={"ID":"36135db2-697e-40d0-9824-57128be4bf3a","Type":"ContainerStarted","Data":"39a8d307315b77b4b466cec70419cd157548e40c3732f856cd5a9589acc2b190"}
Sep 10 14:28:59.080911 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:28:59.080853    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28" podStartSLOduration=1.020052342 podStartE2EDuration="2.080836945s" totalImagesPullingTime="1.060784603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:59.079078084 +0000 UTC m=+557.270606282" watchObservedRunningTime="2026-09-10 14:28:59.080836945 +0000 UTC m=+557.272365142"
Sep 10 14:29:07.337672 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:07.337635    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:29:07.338132 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:07.337696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:29:07.351126 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:07.351096    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:29:08.123207 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:08.123175    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-css28"
Sep 10 14:29:16.718225 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.718194    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"]
Sep 10 14:29:16.726744 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.726714    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.729312 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.729284    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 14:29:16.730918 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.730894    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"]
Sep 10 14:29:16.802403 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.802365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2fe6e5-69e2-4c53-aa13-912642cdab26-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.802580 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.802421    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpm7h\" (UniqueName: \"kubernetes.io/projected/1c2fe6e5-69e2-4c53-aa13-912642cdab26-kube-api-access-hpm7h\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.802580 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.802473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2fe6e5-69e2-4c53-aa13-912642cdab26-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.802580 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.802502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2fe6e5-69e2-4c53-aa13-912642cdab26-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.802580 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.802524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2fe6e5-69e2-4c53-aa13-912642cdab26-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.802580 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.802568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2fe6e5-69e2-4c53-aa13-912642cdab26-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.903811 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.903770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2fe6e5-69e2-4c53-aa13-912642cdab26-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.904014 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.903824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2fe6e5-69e2-4c53-aa13-912642cdab26-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.904014 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.903845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2fe6e5-69e2-4c53-aa13-912642cdab26-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.904014 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.903904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2fe6e5-69e2-4c53-aa13-912642cdab26-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.904014 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.903965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2fe6e5-69e2-4c53-aa13-912642cdab26-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.904222 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.904044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpm7h\" (UniqueName: \"kubernetes.io/projected/1c2fe6e5-69e2-4c53-aa13-912642cdab26-kube-api-access-hpm7h\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.904317 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.904289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2fe6e5-69e2-4c53-aa13-912642cdab26-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.904372 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.904316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2fe6e5-69e2-4c53-aa13-912642cdab26-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.904372 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.904355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2fe6e5-69e2-4c53-aa13-912642cdab26-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.906263 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.906235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2fe6e5-69e2-4c53-aa13-912642cdab26-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.906540 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.906519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2fe6e5-69e2-4c53-aa13-912642cdab26-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:16.912157 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:16.912134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpm7h\" (UniqueName: \"kubernetes.io/projected/1c2fe6e5-69e2-4c53-aa13-912642cdab26-kube-api-access-hpm7h\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m\" (UID: \"1c2fe6e5-69e2-4c53-aa13-912642cdab26\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:17.037768 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:17.037682    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:17.312546 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:17.312518    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"]
Sep 10 14:29:17.314100 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:29:17.314064    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2fe6e5_69e2_4c53_aa13_912642cdab26.slice/crio-b5ab9d4358a15cbcbfba362e15bcf4a74b0b155c67318f90cad4c9bec467c9bf WatchSource:0}: Error finding container b5ab9d4358a15cbcbfba362e15bcf4a74b0b155c67318f90cad4c9bec467c9bf: Status 404 returned error can't find the container with id b5ab9d4358a15cbcbfba362e15bcf4a74b0b155c67318f90cad4c9bec467c9bf
Sep 10 14:29:18.164168 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:18.164128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m" event={"ID":"1c2fe6e5-69e2-4c53-aa13-912642cdab26","Type":"ContainerStarted","Data":"50ac7f2e44b96675faa08657e3370e3e0128b3ff6139d456dc05fc37897df4e4"}
Sep 10 14:29:18.164551 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:18.164226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m" event={"ID":"1c2fe6e5-69e2-4c53-aa13-912642cdab26","Type":"ContainerStarted","Data":"b5ab9d4358a15cbcbfba362e15bcf4a74b0b155c67318f90cad4c9bec467c9bf"}
Sep 10 14:29:18.189508 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:18.189444    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m" podStartSLOduration=1.9458034450000001 podStartE2EDuration="2.189429858s" totalImagesPullingTime="243.626413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:29:18.18548968 +0000 UTC m=+576.377017878" watchObservedRunningTime="2026-09-10 14:29:18.189429858 +0000 UTC m=+576.380958082"
Sep 10 14:29:27.038523 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:27.038476    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:27.039038 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:27.038551    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:27.050668 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:27.050636    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:27.227558 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:27.227533    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m"
Sep 10 14:29:42.252464 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:42.252429    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:29:42.254015 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:42.253991    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:29:54.906219 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:54.906186    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"]
Sep 10 14:29:54.909717 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:54.909695    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:54.912109 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:54.912086    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 14:29:54.918138 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:54.918113    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"]
Sep 10 14:29:54.945836 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:54.945802    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:54.946012 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:54.945852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:54.946012 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:54.945918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:54.946012 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:54.945954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:54.946130 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:54.946020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:54.946130 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:54.946044    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh9sv\" (UniqueName: \"kubernetes.io/projected/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-kube-api-access-sh9sv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:55.046707 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:55.046671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh9sv\" (UniqueName: \"kubernetes.io/projected/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-kube-api-access-sh9sv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:55.046898 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:55.046727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:55.046898 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:55.046768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:55.046898 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:55.046799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:55.046898 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:55.046816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:55.047136 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:55.047019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:55.047365 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:55.047333    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:55.047520 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:55.047408    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:55.047520 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:55.047455    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:55.049270 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:55.049247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:55.049405 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:55.049388    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:55.054540 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:55.054512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh9sv\" (UniqueName: \"kubernetes.io/projected/22de848f-e5a4-4e08-8ae2-7fd2dc425c25-kube-api-access-sh9sv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf\" (UID: \"22de848f-e5a4-4e08-8ae2-7fd2dc425c25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:55.220623 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:55.220529    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:29:55.490586 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:55.490560    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"]
Sep 10 14:29:55.492304 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:29:55.492278    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22de848f_e5a4_4e08_8ae2_7fd2dc425c25.slice/crio-3d0272bb7fb2762f7308944ebd770c91d4ee3f7fc23d0e7c49c775dda313dcb1 WatchSource:0}: Error finding container 3d0272bb7fb2762f7308944ebd770c91d4ee3f7fc23d0e7c49c775dda313dcb1: Status 404 returned error can't find the container with id 3d0272bb7fb2762f7308944ebd770c91d4ee3f7fc23d0e7c49c775dda313dcb1
Sep 10 14:29:55.494089 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:55.494071    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:29:56.368482 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:56.368451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf" event={"ID":"22de848f-e5a4-4e08-8ae2-7fd2dc425c25","Type":"ContainerStarted","Data":"b382865d0ab3f0617a0044b53eef302ef19bb875bb19ac8c19136f2bf7c41bdf"}
Sep 10 14:29:56.368482 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:56.368490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf" event={"ID":"22de848f-e5a4-4e08-8ae2-7fd2dc425c25","Type":"ContainerStarted","Data":"3d0272bb7fb2762f7308944ebd770c91d4ee3f7fc23d0e7c49c775dda313dcb1"}
Sep 10 14:29:56.385548 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:29:56.385499    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf" podStartSLOduration=2.007665608 podStartE2EDuration="2.38548342s" totalImagesPullingTime="377.817812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:29:56.384613892 +0000 UTC m=+614.576142096" watchObservedRunningTime="2026-09-10 14:29:56.38548342 +0000 UTC m=+614.577011618"
Sep 10 14:30:05.221491 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:30:05.221403    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:30:05.221491 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:30:05.221457    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:30:05.233537 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:30:05.233509    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:30:05.427505 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:30:05.427473    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf"
Sep 10 14:31:07.818959 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:07.818918    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cfffdfc98-p6wwb"]
Sep 10 14:31:07.819538 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:07.819151    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6cfffdfc98-p6wwb" podUID="4f27ed99-a646-413e-b610-c7e5265f05ad" containerName="authorino" containerID="cri-o://ae11bd6d164875395856c72f639ce9e0a5b19f727fad78a4f8f552bf9526da92" gracePeriod=30
Sep 10 14:31:08.058482 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.058456    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cfffdfc98-p6wwb"
Sep 10 14:31:08.211088 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.211056    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4f27ed99-a646-413e-b610-c7e5265f05ad-tls-cert\") pod \"4f27ed99-a646-413e-b610-c7e5265f05ad\" (UID: \"4f27ed99-a646-413e-b610-c7e5265f05ad\") "
Sep 10 14:31:08.211253 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.211121    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jjct\" (UniqueName: \"kubernetes.io/projected/4f27ed99-a646-413e-b610-c7e5265f05ad-kube-api-access-2jjct\") pod \"4f27ed99-a646-413e-b610-c7e5265f05ad\" (UID: \"4f27ed99-a646-413e-b610-c7e5265f05ad\") "
Sep 10 14:31:08.213171 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.213134    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f27ed99-a646-413e-b610-c7e5265f05ad-kube-api-access-2jjct" (OuterVolumeSpecName: "kube-api-access-2jjct") pod "4f27ed99-a646-413e-b610-c7e5265f05ad" (UID: "4f27ed99-a646-413e-b610-c7e5265f05ad"). InnerVolumeSpecName "kube-api-access-2jjct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:31:08.221440 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.221416    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f27ed99-a646-413e-b610-c7e5265f05ad-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4f27ed99-a646-413e-b610-c7e5265f05ad" (UID: "4f27ed99-a646-413e-b610-c7e5265f05ad"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:31:08.317944 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.312377    2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4f27ed99-a646-413e-b610-c7e5265f05ad-tls-cert\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:31:08.317944 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.312413    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jjct\" (UniqueName: \"kubernetes.io/projected/4f27ed99-a646-413e-b610-c7e5265f05ad-kube-api-access-2jjct\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:31:08.749295 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.749253    2580 generic.go:356] "Generic (PLEG): container finished" podID="4f27ed99-a646-413e-b610-c7e5265f05ad" containerID="ae11bd6d164875395856c72f639ce9e0a5b19f727fad78a4f8f552bf9526da92" exitCode=0
Sep 10 14:31:08.749463 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.749317    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cfffdfc98-p6wwb"
Sep 10 14:31:08.749463 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.749321    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cfffdfc98-p6wwb" event={"ID":"4f27ed99-a646-413e-b610-c7e5265f05ad","Type":"ContainerDied","Data":"ae11bd6d164875395856c72f639ce9e0a5b19f727fad78a4f8f552bf9526da92"}
Sep 10 14:31:08.749463 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.749359    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cfffdfc98-p6wwb" event={"ID":"4f27ed99-a646-413e-b610-c7e5265f05ad","Type":"ContainerDied","Data":"d5f7af95e946f621b62d7f7d84d5b9602c5bb1f272c955974bf481a8bbd04771"}
Sep 10 14:31:08.749463 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.749381    2580 scope.go:117] "RemoveContainer" containerID="ae11bd6d164875395856c72f639ce9e0a5b19f727fad78a4f8f552bf9526da92"
Sep 10 14:31:08.758398 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.758380    2580 scope.go:117] "RemoveContainer" containerID="ae11bd6d164875395856c72f639ce9e0a5b19f727fad78a4f8f552bf9526da92"
Sep 10 14:31:08.758673 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:31:08.758653    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae11bd6d164875395856c72f639ce9e0a5b19f727fad78a4f8f552bf9526da92\": container with ID starting with ae11bd6d164875395856c72f639ce9e0a5b19f727fad78a4f8f552bf9526da92 not found: ID does not exist" containerID="ae11bd6d164875395856c72f639ce9e0a5b19f727fad78a4f8f552bf9526da92"
Sep 10 14:31:08.758737 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.758686    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae11bd6d164875395856c72f639ce9e0a5b19f727fad78a4f8f552bf9526da92"} err="failed to get container status \"ae11bd6d164875395856c72f639ce9e0a5b19f727fad78a4f8f552bf9526da92\": rpc error: code = NotFound desc = could not find container \"ae11bd6d164875395856c72f639ce9e0a5b19f727fad78a4f8f552bf9526da92\": container with ID starting with ae11bd6d164875395856c72f639ce9e0a5b19f727fad78a4f8f552bf9526da92 not found: ID does not exist"
Sep 10 14:31:08.767591 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.767570    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cfffdfc98-p6wwb"]
Sep 10 14:31:08.769645 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:08.769625    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6cfffdfc98-p6wwb"]
Sep 10 14:31:10.309739 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:10.309707    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f27ed99-a646-413e-b610-c7e5265f05ad" path="/var/lib/kubelet/pods/4f27ed99-a646-413e-b610-c7e5265f05ad/volumes"
Sep 10 14:31:26.900358 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:26.900320    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7943-63da81"]
Sep 10 14:31:26.900727 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:26.900709    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f27ed99-a646-413e-b610-c7e5265f05ad" containerName="authorino"
Sep 10 14:31:26.900727 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:26.900721    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f27ed99-a646-413e-b610-c7e5265f05ad" containerName="authorino"
Sep 10 14:31:26.900797 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:26.900792    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f27ed99-a646-413e-b610-c7e5265f05ad" containerName="authorino"
Sep 10 14:31:26.905313 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:26.905295    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-63da81"
Sep 10 14:31:26.908205 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:26.908183    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qjcz8\""
Sep 10 14:31:26.913596 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:26.913574    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7943-63da81"]
Sep 10 14:31:26.971812 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:26.971768    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lhx8\" (UniqueName: \"kubernetes.io/projected/a294e08f-3332-4f2a-8acb-d64dd2765263-kube-api-access-7lhx8\") pod \"test-curl-7943-63da81\" (UID: \"a294e08f-3332-4f2a-8acb-d64dd2765263\") " pod="opendatahub/test-curl-7943-63da81"
Sep 10 14:31:27.072402 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:27.072370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lhx8\" (UniqueName: \"kubernetes.io/projected/a294e08f-3332-4f2a-8acb-d64dd2765263-kube-api-access-7lhx8\") pod \"test-curl-7943-63da81\" (UID: \"a294e08f-3332-4f2a-8acb-d64dd2765263\") " pod="opendatahub/test-curl-7943-63da81"
Sep 10 14:31:27.081564 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:27.081529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lhx8\" (UniqueName: \"kubernetes.io/projected/a294e08f-3332-4f2a-8acb-d64dd2765263-kube-api-access-7lhx8\") pod \"test-curl-7943-63da81\" (UID: \"a294e08f-3332-4f2a-8acb-d64dd2765263\") " pod="opendatahub/test-curl-7943-63da81"
Sep 10 14:31:27.214746 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:27.214652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-63da81"
Sep 10 14:31:27.487033 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:27.487002    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7943-63da81"]
Sep 10 14:31:27.488967 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:31:27.488938    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda294e08f_3332_4f2a_8acb_d64dd2765263.slice/crio-473bd9e717a206fa11c0f0e7598ff8e825260d4b67b9f98bf64c1795401be13d WatchSource:0}: Error finding container 473bd9e717a206fa11c0f0e7598ff8e825260d4b67b9f98bf64c1795401be13d: Status 404 returned error can't find the container with id 473bd9e717a206fa11c0f0e7598ff8e825260d4b67b9f98bf64c1795401be13d
Sep 10 14:31:27.850670 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:27.850587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7943-63da81" event={"ID":"a294e08f-3332-4f2a-8acb-d64dd2765263","Type":"ContainerStarted","Data":"473bd9e717a206fa11c0f0e7598ff8e825260d4b67b9f98bf64c1795401be13d"}
Sep 10 14:31:29.644850 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:29.644814    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"]
Sep 10 14:31:29.656163 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:29.656134    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:29.659297 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:29.659048    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-600197-serving-cert\""
Sep 10 14:31:29.662862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:29.662367    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"]
Sep 10 14:31:29.693214 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:29.693178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98aad721-07cf-460e-b480-c72053cc063d-maas-api-tls\") pod \"maas-api-e2e-inf-a-600197-684694cf78-9l5zc\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:29.693405 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:29.693227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98aad721-07cf-460e-b480-c72053cc063d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-600197-684694cf78-9l5zc\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:29.693405 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:29.693316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98aad721-07cf-460e-b480-c72053cc063d-metrics-certs\") pod \"maas-api-e2e-inf-a-600197-684694cf78-9l5zc\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:29.693405 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:29.693348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll6cm\" (UniqueName: \"kubernetes.io/projected/98aad721-07cf-460e-b480-c72053cc063d-kube-api-access-ll6cm\") pod \"maas-api-e2e-inf-a-600197-684694cf78-9l5zc\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:29.794023 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:29.793957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98aad721-07cf-460e-b480-c72053cc063d-maas-api-tls\") pod \"maas-api-e2e-inf-a-600197-684694cf78-9l5zc\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:29.794023 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:29.794028    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98aad721-07cf-460e-b480-c72053cc063d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-600197-684694cf78-9l5zc\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:29.794291 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:29.794107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98aad721-07cf-460e-b480-c72053cc063d-metrics-certs\") pod \"maas-api-e2e-inf-a-600197-684694cf78-9l5zc\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:29.794291 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:29.794137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll6cm\" (UniqueName: \"kubernetes.io/projected/98aad721-07cf-460e-b480-c72053cc063d-kube-api-access-ll6cm\") pod \"maas-api-e2e-inf-a-600197-684694cf78-9l5zc\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:29.794291 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:31:29.794140    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-serving-cert: secret "maas-api-e2e-inf-a-600197-serving-cert" not found
Sep 10 14:31:29.794484 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:31:29.794324    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98aad721-07cf-460e-b480-c72053cc063d-maas-api-tls podName:98aad721-07cf-460e-b480-c72053cc063d nodeName:}" failed. No retries permitted until 2026-09-10 14:31:30.29419817 +0000 UTC m=+708.485726351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98aad721-07cf-460e-b480-c72053cc063d-maas-api-tls") pod "maas-api-e2e-inf-a-600197-684694cf78-9l5zc" (UID: "98aad721-07cf-460e-b480-c72053cc063d") : secret "maas-api-e2e-inf-a-600197-serving-cert" not found
Sep 10 14:31:29.794874 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:29.794841    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98aad721-07cf-460e-b480-c72053cc063d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-600197-684694cf78-9l5zc\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:29.797472 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:29.797449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98aad721-07cf-460e-b480-c72053cc063d-metrics-certs\") pod \"maas-api-e2e-inf-a-600197-684694cf78-9l5zc\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:29.806583 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:29.806549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll6cm\" (UniqueName: \"kubernetes.io/projected/98aad721-07cf-460e-b480-c72053cc063d-kube-api-access-ll6cm\") pod \"maas-api-e2e-inf-a-600197-684694cf78-9l5zc\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:30.297600 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:30.297501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98aad721-07cf-460e-b480-c72053cc063d-maas-api-tls\") pod \"maas-api-e2e-inf-a-600197-684694cf78-9l5zc\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:30.299887 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:30.299856    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98aad721-07cf-460e-b480-c72053cc063d-maas-api-tls\") pod \"maas-api-e2e-inf-a-600197-684694cf78-9l5zc\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:30.574174 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:30.574087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:30.853708 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:30.853679    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"]
Sep 10 14:31:30.855331 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:31:30.855292    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98aad721_07cf_460e_b480_c72053cc063d.slice/crio-019c286e4d9f5604e361492b0a3a46b2f537a647a57f2dc13541c536273fc7d6 WatchSource:0}: Error finding container 019c286e4d9f5604e361492b0a3a46b2f537a647a57f2dc13541c536273fc7d6: Status 404 returned error can't find the container with id 019c286e4d9f5604e361492b0a3a46b2f537a647a57f2dc13541c536273fc7d6
Sep 10 14:31:30.874136 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:30.874103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7943-63da81" event={"ID":"a294e08f-3332-4f2a-8acb-d64dd2765263","Type":"ContainerStarted","Data":"073b4a09da4f1e2343fd65425b43e530c5993b468cb52889b1b82889d35fa1af"}
Sep 10 14:31:30.875430 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:30.875408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc" event={"ID":"98aad721-07cf-460e-b480-c72053cc063d","Type":"ContainerStarted","Data":"019c286e4d9f5604e361492b0a3a46b2f537a647a57f2dc13541c536273fc7d6"}
Sep 10 14:31:31.507311 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.507256    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7943-63da81" podStartSLOduration=3.157707916 podStartE2EDuration="5.507235202s" totalImagesPullingTime="2.349527286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:31:30.889693119 +0000 UTC m=+709.081221320" watchObservedRunningTime="2026-09-10 14:31:31.507235202 +0000 UTC m=+709.698763417"
Sep 10 14:31:31.508206 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.508184    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7943-63da81"]
Sep 10 14:31:31.516597 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.516567    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7943-63da81"]
Sep 10 14:31:31.630611 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.630537    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7943-3906fb"]
Sep 10 14:31:31.630923 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.630910    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a294e08f-3332-4f2a-8acb-d64dd2765263" containerName="test-curl-7943-63da81"
Sep 10 14:31:31.631014 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.630924    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a294e08f-3332-4f2a-8acb-d64dd2765263" containerName="test-curl-7943-63da81"
Sep 10 14:31:31.631060 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.631023    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a294e08f-3332-4f2a-8acb-d64dd2765263" containerName="test-curl-7943-63da81"
Sep 10 14:31:31.634171 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.634156    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-3906fb"
Sep 10 14:31:31.642339 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.642314    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7943-3906fb"]
Sep 10 14:31:31.708707 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.708671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h69lg\" (UniqueName: \"kubernetes.io/projected/55ef42b8-ec6a-4645-9050-7bf114cb304b-kube-api-access-h69lg\") pod \"test-curl-7943-3906fb\" (UID: \"55ef42b8-ec6a-4645-9050-7bf114cb304b\") " pod="opendatahub/test-curl-7943-3906fb"
Sep 10 14:31:31.809598 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.809560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h69lg\" (UniqueName: \"kubernetes.io/projected/55ef42b8-ec6a-4645-9050-7bf114cb304b-kube-api-access-h69lg\") pod \"test-curl-7943-3906fb\" (UID: \"55ef42b8-ec6a-4645-9050-7bf114cb304b\") " pod="opendatahub/test-curl-7943-3906fb"
Sep 10 14:31:31.818678 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.818647    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h69lg\" (UniqueName: \"kubernetes.io/projected/55ef42b8-ec6a-4645-9050-7bf114cb304b-kube-api-access-h69lg\") pod \"test-curl-7943-3906fb\" (UID: \"55ef42b8-ec6a-4645-9050-7bf114cb304b\") " pod="opendatahub/test-curl-7943-3906fb"
Sep 10 14:31:31.881764 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.881736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc" event={"ID":"98aad721-07cf-460e-b480-c72053cc063d","Type":"ContainerStarted","Data":"04339c14ad255e1dc1334b010e56b0659111a06b832ce9183b3dcfdd1a6984f7"}
Sep 10 14:31:31.882190 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:31:31.881928    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7943-63da81" container="test-curl-7943-63da81"
Sep 10 14:31:31.882190 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.881981    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:31.902926 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.902882    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc" podStartSLOduration=2.480982779 podStartE2EDuration="2.902865864s" totalImagesPullingTime="421.883085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:31:31.901320264 +0000 UTC m=+710.092848461" watchObservedRunningTime="2026-09-10 14:31:31.902865864 +0000 UTC m=+710.094394119"
Sep 10 14:31:31.944136 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:31.944090    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-3906fb"
Sep 10 14:31:32.217727 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:32.217700    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7943-3906fb"]
Sep 10 14:31:32.219617 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:31:32.219586    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55ef42b8_ec6a_4645_9050_7bf114cb304b.slice/crio-bf5ab9ef206bd0c6add48aa1612279d2eb4b43a31a3522fa66b0bc5b6e500810 WatchSource:0}: Error finding container bf5ab9ef206bd0c6add48aa1612279d2eb4b43a31a3522fa66b0bc5b6e500810: Status 404 returned error can't find the container with id bf5ab9ef206bd0c6add48aa1612279d2eb4b43a31a3522fa66b0bc5b6e500810
Sep 10 14:31:32.888557 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:32.888514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7943-3906fb" event={"ID":"55ef42b8-ec6a-4645-9050-7bf114cb304b","Type":"ContainerStarted","Data":"c7963624d3827b0bce7105f684e0323ffe7233ebcb0f8878342a21331b0ced64"}
Sep 10 14:31:32.888557 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:32.888560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7943-3906fb" event={"ID":"55ef42b8-ec6a-4645-9050-7bf114cb304b","Type":"ContainerStarted","Data":"bf5ab9ef206bd0c6add48aa1612279d2eb4b43a31a3522fa66b0bc5b6e500810"}
Sep 10 14:31:32.889064 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:32.888791    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7943-63da81" podUID="a294e08f-3332-4f2a-8acb-d64dd2765263" containerName="test-curl-7943-63da81" containerID="cri-o://073b4a09da4f1e2343fd65425b43e530c5993b468cb52889b1b82889d35fa1af" gracePeriod=2
Sep 10 14:31:32.905280 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:32.905235    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7943-3906fb" podStartSLOduration=1.905220598 podStartE2EDuration="1.905220598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:31:32.902842054 +0000 UTC m=+711.094370251" watchObservedRunningTime="2026-09-10 14:31:32.905220598 +0000 UTC m=+711.096748797"
Sep 10 14:31:33.532605 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:33.532518    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7943-3906fb"]
Sep 10 14:31:33.540567 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:33.540537    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7943-3906fb"]
Sep 10 14:31:33.895423 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:31:33.895394    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7943-3906fb" container="test-curl-7943-3906fb"
Sep 10 14:31:34.900144 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:34.900108    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7943-3906fb" podUID="55ef42b8-ec6a-4645-9050-7bf114cb304b" containerName="test-curl-7943-3906fb" containerID="cri-o://c7963624d3827b0bce7105f684e0323ffe7233ebcb0f8878342a21331b0ced64" gracePeriod=2
Sep 10 14:31:35.075096 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:35.075068    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-63da81"
Sep 10 14:31:35.144189 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:35.144100    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lhx8\" (UniqueName: \"kubernetes.io/projected/a294e08f-3332-4f2a-8acb-d64dd2765263-kube-api-access-7lhx8\") pod \"a294e08f-3332-4f2a-8acb-d64dd2765263\" (UID: \"a294e08f-3332-4f2a-8acb-d64dd2765263\") "
Sep 10 14:31:35.146115 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:35.146087    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a294e08f-3332-4f2a-8acb-d64dd2765263-kube-api-access-7lhx8" (OuterVolumeSpecName: "kube-api-access-7lhx8") pod "a294e08f-3332-4f2a-8acb-d64dd2765263" (UID: "a294e08f-3332-4f2a-8acb-d64dd2765263"). InnerVolumeSpecName "kube-api-access-7lhx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:31:35.245693 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:35.245663    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lhx8\" (UniqueName: \"kubernetes.io/projected/a294e08f-3332-4f2a-8acb-d64dd2765263-kube-api-access-7lhx8\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:31:35.906631 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:35.906597    2580 generic.go:356] "Generic (PLEG): container finished" podID="a294e08f-3332-4f2a-8acb-d64dd2765263" containerID="073b4a09da4f1e2343fd65425b43e530c5993b468cb52889b1b82889d35fa1af" exitCode=137
Sep 10 14:31:35.907054 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:35.906639    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-63da81"
Sep 10 14:31:35.907054 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:35.906705    2580 scope.go:117] "RemoveContainer" containerID="073b4a09da4f1e2343fd65425b43e530c5993b468cb52889b1b82889d35fa1af"
Sep 10 14:31:35.915556 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:35.915538    2580 scope.go:117] "RemoveContainer" containerID="073b4a09da4f1e2343fd65425b43e530c5993b468cb52889b1b82889d35fa1af"
Sep 10 14:31:35.915827 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:31:35.915811    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073b4a09da4f1e2343fd65425b43e530c5993b468cb52889b1b82889d35fa1af\": container with ID starting with 073b4a09da4f1e2343fd65425b43e530c5993b468cb52889b1b82889d35fa1af not found: ID does not exist" containerID="073b4a09da4f1e2343fd65425b43e530c5993b468cb52889b1b82889d35fa1af"
Sep 10 14:31:35.915881 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:35.915842    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073b4a09da4f1e2343fd65425b43e530c5993b468cb52889b1b82889d35fa1af"} err="failed to get container status \"073b4a09da4f1e2343fd65425b43e530c5993b468cb52889b1b82889d35fa1af\": rpc error: code = NotFound desc = could not find container \"073b4a09da4f1e2343fd65425b43e530c5993b468cb52889b1b82889d35fa1af\": container with ID starting with 073b4a09da4f1e2343fd65425b43e530c5993b468cb52889b1b82889d35fa1af not found: ID does not exist"
Sep 10 14:31:35.916616 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:31:35.916593    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7943-63da81\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="a294e08f-3332-4f2a-8acb-d64dd2765263" pod="opendatahub/test-curl-7943-63da81"
Sep 10 14:31:36.309955 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:36.309854    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a294e08f-3332-4f2a-8acb-d64dd2765263" path="/var/lib/kubelet/pods/a294e08f-3332-4f2a-8acb-d64dd2765263/volumes"
Sep 10 14:31:37.077458 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.077418    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-3906fb"
Sep 10 14:31:37.163298 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.163210    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h69lg\" (UniqueName: \"kubernetes.io/projected/55ef42b8-ec6a-4645-9050-7bf114cb304b-kube-api-access-h69lg\") pod \"55ef42b8-ec6a-4645-9050-7bf114cb304b\" (UID: \"55ef42b8-ec6a-4645-9050-7bf114cb304b\") "
Sep 10 14:31:37.165278 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.165244    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ef42b8-ec6a-4645-9050-7bf114cb304b-kube-api-access-h69lg" (OuterVolumeSpecName: "kube-api-access-h69lg") pod "55ef42b8-ec6a-4645-9050-7bf114cb304b" (UID: "55ef42b8-ec6a-4645-9050-7bf114cb304b"). InnerVolumeSpecName "kube-api-access-h69lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:31:37.264184 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.264147    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h69lg\" (UniqueName: \"kubernetes.io/projected/55ef42b8-ec6a-4645-9050-7bf114cb304b-kube-api-access-h69lg\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:31:37.543471 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.543396    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"]
Sep 10 14:31:37.543810 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.543797    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ef42b8-ec6a-4645-9050-7bf114cb304b" containerName="test-curl-7943-3906fb"
Sep 10 14:31:37.543854 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.543812    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ef42b8-ec6a-4645-9050-7bf114cb304b" containerName="test-curl-7943-3906fb"
Sep 10 14:31:37.543909 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.543885    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="55ef42b8-ec6a-4645-9050-7bf114cb304b" containerName="test-curl-7943-3906fb"
Sep 10 14:31:37.546905 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.546881    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:37.550335 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.550315    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ad4f3b47-serving-cert\""
Sep 10 14:31:37.563060 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.563037    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"]
Sep 10 14:31:37.567880 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.567848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmv8p\" (UniqueName: \"kubernetes.io/projected/b6feac28-ede3-460c-83eb-b7622f8b562d-kube-api-access-gmv8p\") pod \"maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:37.568006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.567901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6feac28-ede3-460c-83eb-b7622f8b562d-maas-api-tls\") pod \"maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:37.568137 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.568013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6feac28-ede3-460c-83eb-b7622f8b562d-metrics-certs\") pod \"maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:37.568137 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.568057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6feac28-ede3-460c-83eb-b7622f8b562d-openshift-service-ca\") pod \"maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:37.669384 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.669348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6feac28-ede3-460c-83eb-b7622f8b562d-metrics-certs\") pod \"maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:37.669566 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.669397    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6feac28-ede3-460c-83eb-b7622f8b562d-openshift-service-ca\") pod \"maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:37.669566 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.669470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmv8p\" (UniqueName: \"kubernetes.io/projected/b6feac28-ede3-460c-83eb-b7622f8b562d-kube-api-access-gmv8p\") pod \"maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:37.669566 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.669501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6feac28-ede3-460c-83eb-b7622f8b562d-maas-api-tls\") pod \"maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:37.669755 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:31:37.669586    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-serving-cert: secret "maas-api-e2e-ait-ad4f3b47-serving-cert" not found
Sep 10 14:31:37.669755 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:31:37.669651    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6feac28-ede3-460c-83eb-b7622f8b562d-maas-api-tls podName:b6feac28-ede3-460c-83eb-b7622f8b562d nodeName:}" failed. No retries permitted until 2026-09-10 14:31:38.169630892 +0000 UTC m=+716.361159089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b6feac28-ede3-460c-83eb-b7622f8b562d-maas-api-tls") pod "maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s" (UID: "b6feac28-ede3-460c-83eb-b7622f8b562d") : secret "maas-api-e2e-ait-ad4f3b47-serving-cert" not found
Sep 10 14:31:37.670197 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.670176    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6feac28-ede3-460c-83eb-b7622f8b562d-openshift-service-ca\") pod \"maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:37.671805 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.671784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6feac28-ede3-460c-83eb-b7622f8b562d-metrics-certs\") pod \"maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:37.684123 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.684095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmv8p\" (UniqueName: \"kubernetes.io/projected/b6feac28-ede3-460c-83eb-b7622f8b562d-kube-api-access-gmv8p\") pod \"maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:37.894657 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.894629    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:31:37.920918 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.920883    2580 generic.go:356] "Generic (PLEG): container finished" podID="55ef42b8-ec6a-4645-9050-7bf114cb304b" containerID="c7963624d3827b0bce7105f684e0323ffe7233ebcb0f8878342a21331b0ced64" exitCode=137
Sep 10 14:31:37.920918 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.920918    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-3906fb"
Sep 10 14:31:37.921164 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.920993    2580 scope.go:117] "RemoveContainer" containerID="c7963624d3827b0bce7105f684e0323ffe7233ebcb0f8878342a21331b0ced64"
Sep 10 14:31:37.930401 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.930387    2580 scope.go:117] "RemoveContainer" containerID="c7963624d3827b0bce7105f684e0323ffe7233ebcb0f8878342a21331b0ced64"
Sep 10 14:31:37.930659 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:31:37.930644    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7963624d3827b0bce7105f684e0323ffe7233ebcb0f8878342a21331b0ced64\": container with ID starting with c7963624d3827b0bce7105f684e0323ffe7233ebcb0f8878342a21331b0ced64 not found: ID does not exist" containerID="c7963624d3827b0bce7105f684e0323ffe7233ebcb0f8878342a21331b0ced64"
Sep 10 14:31:37.930701 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:37.930670    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7963624d3827b0bce7105f684e0323ffe7233ebcb0f8878342a21331b0ced64"} err="failed to get container status \"c7963624d3827b0bce7105f684e0323ffe7233ebcb0f8878342a21331b0ced64\": rpc error: code = NotFound desc = could not find container \"c7963624d3827b0bce7105f684e0323ffe7233ebcb0f8878342a21331b0ced64\": container with ID starting with c7963624d3827b0bce7105f684e0323ffe7233ebcb0f8878342a21331b0ced64 not found: ID does not exist"
Sep 10 14:31:37.932076 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:31:37.932058    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7943-3906fb\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="55ef42b8-ec6a-4645-9050-7bf114cb304b" pod="opendatahub/test-curl-7943-3906fb"
Sep 10 14:31:38.175524 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:38.175441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6feac28-ede3-460c-83eb-b7622f8b562d-maas-api-tls\") pod \"maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:38.177956 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:38.177938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6feac28-ede3-460c-83eb-b7622f8b562d-maas-api-tls\") pod \"maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:38.320069 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:38.320026    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55ef42b8-ec6a-4645-9050-7bf114cb304b" path="/var/lib/kubelet/pods/55ef42b8-ec6a-4645-9050-7bf114cb304b/volumes"
Sep 10 14:31:38.457197 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:38.457111    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:38.737596 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:38.737534    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"]
Sep 10 14:31:38.740059 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:31:38.740030    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6feac28_ede3_460c_83eb_b7622f8b562d.slice/crio-943238dd0b8c901565f8fe1a7843fd8f98c5e7e9c8f5dd4a9ccc5500bb1f5925 WatchSource:0}: Error finding container 943238dd0b8c901565f8fe1a7843fd8f98c5e7e9c8f5dd4a9ccc5500bb1f5925: Status 404 returned error can't find the container with id 943238dd0b8c901565f8fe1a7843fd8f98c5e7e9c8f5dd4a9ccc5500bb1f5925
Sep 10 14:31:38.928644 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:38.928611    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s" event={"ID":"b6feac28-ede3-460c-83eb-b7622f8b562d","Type":"ContainerStarted","Data":"943238dd0b8c901565f8fe1a7843fd8f98c5e7e9c8f5dd4a9ccc5500bb1f5925"}
Sep 10 14:31:39.397161 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.397125    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"]
Sep 10 14:31:39.400851 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.400819    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:39.403524 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.403499    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-d24d4c-serving-cert\""
Sep 10 14:31:39.413029 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.412947    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"]
Sep 10 14:31:39.488159 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.488127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-maas-api-tls\") pod \"maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:39.488159 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.488159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-openshift-service-ca\") pod \"maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:39.488404 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.488182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcxp4\" (UniqueName: \"kubernetes.io/projected/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-kube-api-access-rcxp4\") pod \"maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:39.488404 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.488241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-metrics-certs\") pod \"maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:39.589245 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.589209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-maas-api-tls\") pod \"maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:39.589245 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.589247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-openshift-service-ca\") pod \"maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:39.589482 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.589267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcxp4\" (UniqueName: \"kubernetes.io/projected/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-kube-api-access-rcxp4\") pod \"maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:39.589482 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.589294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-metrics-certs\") pod \"maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:39.589482 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:31:39.589377    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-serving-cert: secret "maas-api-e2e-autores-d24d4c-serving-cert" not found
Sep 10 14:31:39.589482 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:31:39.589468    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-maas-api-tls podName:2e1bc717-9c1a-4e73-a6f0-246bbb5994b4 nodeName:}" failed. No retries permitted until 2026-09-10 14:31:40.089434964 +0000 UTC m=+718.280963154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-maas-api-tls") pod "maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z" (UID: "2e1bc717-9c1a-4e73-a6f0-246bbb5994b4") : secret "maas-api-e2e-autores-d24d4c-serving-cert" not found
Sep 10 14:31:39.589909 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.589891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-openshift-service-ca\") pod \"maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:39.591670 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.591651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-metrics-certs\") pod \"maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:39.598604 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.598577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcxp4\" (UniqueName: \"kubernetes.io/projected/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-kube-api-access-rcxp4\") pod \"maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:39.936024 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.935966    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s" event={"ID":"b6feac28-ede3-460c-83eb-b7622f8b562d","Type":"ContainerStarted","Data":"4662f61f545d710563265229d3eb9e0154e6aa70064d78e8171a0ea192274453"}
Sep 10 14:31:39.936207 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.936095    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:39.963025 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:39.962954    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s" podStartSLOduration=2.688757646 podStartE2EDuration="2.962937309s" totalImagesPullingTime="274.179663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:31:39.961041537 +0000 UTC m=+718.152569735" watchObservedRunningTime="2026-09-10 14:31:39.962937309 +0000 UTC m=+718.154465507"
Sep 10 14:31:40.094155 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.094119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-maas-api-tls\") pod \"maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:40.096526 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.096501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-maas-api-tls\") pod \"maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:40.267271 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.267189    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"]
Sep 10 14:31:40.270943 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.270923    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"
Sep 10 14:31:40.274355 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.274332    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-d24d4c-dockercfg-9xz7p\""
Sep 10 14:31:40.275005 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.274985    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-d24d4c\""
Sep 10 14:31:40.284285 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.284261    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"]
Sep 10 14:31:40.311797 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.311771    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:40.396140 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.396107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb4c43af-fdbb-44f3-9ae2-e495bc5842ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq\" (UID: \"fb4c43af-fdbb-44f3-9ae2-e495bc5842ed\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"
Sep 10 14:31:40.396309 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.396280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kz4j\" (UniqueName: \"kubernetes.io/projected/fb4c43af-fdbb-44f3-9ae2-e495bc5842ed-kube-api-access-5kz4j\") pod \"payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq\" (UID: \"fb4c43af-fdbb-44f3-9ae2-e495bc5842ed\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"
Sep 10 14:31:40.496961 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.496922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kz4j\" (UniqueName: \"kubernetes.io/projected/fb4c43af-fdbb-44f3-9ae2-e495bc5842ed-kube-api-access-5kz4j\") pod \"payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq\" (UID: \"fb4c43af-fdbb-44f3-9ae2-e495bc5842ed\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"
Sep 10 14:31:40.496961 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.496986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb4c43af-fdbb-44f3-9ae2-e495bc5842ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq\" (UID: \"fb4c43af-fdbb-44f3-9ae2-e495bc5842ed\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"
Sep 10 14:31:40.497739 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.497717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb4c43af-fdbb-44f3-9ae2-e495bc5842ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq\" (UID: \"fb4c43af-fdbb-44f3-9ae2-e495bc5842ed\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"
Sep 10 14:31:40.506463 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.506430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kz4j\" (UniqueName: \"kubernetes.io/projected/fb4c43af-fdbb-44f3-9ae2-e495bc5842ed-kube-api-access-5kz4j\") pod \"payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq\" (UID: \"fb4c43af-fdbb-44f3-9ae2-e495bc5842ed\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"
Sep 10 14:31:40.582100 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.582015    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"
Sep 10 14:31:40.590374 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.590350    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"]
Sep 10 14:31:40.593284 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:31:40.593252    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1bc717_9c1a_4e73_a6f0_246bbb5994b4.slice/crio-cfcc08f366539ef4c9dbe8e9f5428b7f73803fa05d661e0507ad3382bac730ab WatchSource:0}: Error finding container cfcc08f366539ef4c9dbe8e9f5428b7f73803fa05d661e0507ad3382bac730ab: Status 404 returned error can't find the container with id cfcc08f366539ef4c9dbe8e9f5428b7f73803fa05d661e0507ad3382bac730ab
Sep 10 14:31:40.873558 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.873530    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"]
Sep 10 14:31:40.875262 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:31:40.875232    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb4c43af_fdbb_44f3_9ae2_e495bc5842ed.slice/crio-84c6bfe26e8d3eb736e022c07e4fc5390c0f5d1ae0e62996b9da40fef698ba9f WatchSource:0}: Error finding container 84c6bfe26e8d3eb736e022c07e4fc5390c0f5d1ae0e62996b9da40fef698ba9f: Status 404 returned error can't find the container with id 84c6bfe26e8d3eb736e022c07e4fc5390c0f5d1ae0e62996b9da40fef698ba9f
Sep 10 14:31:40.943654 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.943614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq" event={"ID":"fb4c43af-fdbb-44f3-9ae2-e495bc5842ed","Type":"ContainerStarted","Data":"84c6bfe26e8d3eb736e022c07e4fc5390c0f5d1ae0e62996b9da40fef698ba9f"}
Sep 10 14:31:40.944855 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:40.944829    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z" event={"ID":"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4","Type":"ContainerStarted","Data":"cfcc08f366539ef4c9dbe8e9f5428b7f73803fa05d661e0507ad3382bac730ab"}
Sep 10 14:31:41.951958 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:41.951916    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z" event={"ID":"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4","Type":"ContainerStarted","Data":"b916b2155e2cec042af277eb4123305b062c1d0769df2e4b32a50afca39fa1b0"}
Sep 10 14:31:41.952404 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:41.952037    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:41.970623 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:41.970570    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z" podStartSLOduration=2.283092197 podStartE2EDuration="2.970553647s" totalImagesPullingTime="687.46145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:31:41.968877439 +0000 UTC m=+720.160405644" watchObservedRunningTime="2026-09-10 14:31:41.970553647 +0000 UTC m=+720.162081883"
Sep 10 14:31:43.966536 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:43.966495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq" event={"ID":"fb4c43af-fdbb-44f3-9ae2-e495bc5842ed","Type":"ContainerStarted","Data":"e437b2475bd9c7fe2fbb2b3d1d671c30003a880ad82aaffc487bd50a82911222"}
Sep 10 14:31:43.967051 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:43.966595    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"
Sep 10 14:31:43.996748 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:43.996699    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq" podStartSLOduration=1.775742552 podStartE2EDuration="3.996679884s" totalImagesPullingTime="2.220937332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:31:43.995407783 +0000 UTC m=+722.186935982" watchObservedRunningTime="2026-09-10 14:31:43.996679884 +0000 UTC m=+722.188208083"
Sep 10 14:31:45.951118 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:45.951084    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:31:47.963398 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:47.963368    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:31:54.974850 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:54.974820    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"
Sep 10 14:31:58.516402 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:58.516363    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"]
Sep 10 14:31:58.521136 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:58.521102    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:31:58.526731 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:58.526704    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8ee208-serving-cert\""
Sep 10 14:31:58.562825 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:58.562792    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"]
Sep 10 14:31:58.668295 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:58.668257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmpx5\" (UniqueName: \"kubernetes.io/projected/c8c37460-c57f-4fb2-801d-a559b111d236-kube-api-access-cmpx5\") pod \"maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:31:58.668295 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:58.668296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8c37460-c57f-4fb2-801d-a559b111d236-metrics-certs\") pod \"maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:31:58.668516 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:58.668388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c37460-c57f-4fb2-801d-a559b111d236-maas-api-tls\") pod \"maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:31:58.668516 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:58.668411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c37460-c57f-4fb2-801d-a559b111d236-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:31:58.769119 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:58.769020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c37460-c57f-4fb2-801d-a559b111d236-maas-api-tls\") pod \"maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:31:58.769119 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:58.769078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c37460-c57f-4fb2-801d-a559b111d236-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:31:58.769369 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:31:58.769134    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-serving-cert: secret "maas-api-e2e-inf-b-8ee208-serving-cert" not found
Sep 10 14:31:58.769369 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:31:58.769242    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8c37460-c57f-4fb2-801d-a559b111d236-maas-api-tls podName:c8c37460-c57f-4fb2-801d-a559b111d236 nodeName:}" failed. No retries permitted until 2026-09-10 14:31:59.269220264 +0000 UTC m=+737.460748461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8c37460-c57f-4fb2-801d-a559b111d236-maas-api-tls") pod "maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n" (UID: "c8c37460-c57f-4fb2-801d-a559b111d236") : secret "maas-api-e2e-inf-b-8ee208-serving-cert" not found
Sep 10 14:31:58.769369 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:58.769267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmpx5\" (UniqueName: \"kubernetes.io/projected/c8c37460-c57f-4fb2-801d-a559b111d236-kube-api-access-cmpx5\") pod \"maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:31:58.769369 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:58.769303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8c37460-c57f-4fb2-801d-a559b111d236-metrics-certs\") pod \"maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:31:58.769924 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:58.769900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c37460-c57f-4fb2-801d-a559b111d236-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:31:58.771942 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:58.771922    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8c37460-c57f-4fb2-801d-a559b111d236-metrics-certs\") pod \"maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:31:58.787281 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:58.787253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmpx5\" (UniqueName: \"kubernetes.io/projected/c8c37460-c57f-4fb2-801d-a559b111d236-kube-api-access-cmpx5\") pod \"maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:31:59.275041 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:59.275001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c37460-c57f-4fb2-801d-a559b111d236-maas-api-tls\") pod \"maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:31:59.277611 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:59.277583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c37460-c57f-4fb2-801d-a559b111d236-maas-api-tls\") pod \"maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:31:59.432170 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:59.432131    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:31:59.712345 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:31:59.712317    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"]
Sep 10 14:31:59.714151 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:31:59.714124    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c37460_c57f_4fb2_801d_a559b111d236.slice/crio-e1e1031986f0b0e9cbdf1c667869ad09e48f6a959b6f40bb2605de8c06fead77 WatchSource:0}: Error finding container e1e1031986f0b0e9cbdf1c667869ad09e48f6a959b6f40bb2605de8c06fead77: Status 404 returned error can't find the container with id e1e1031986f0b0e9cbdf1c667869ad09e48f6a959b6f40bb2605de8c06fead77
Sep 10 14:32:00.057411 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:00.057331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n" event={"ID":"c8c37460-c57f-4fb2-801d-a559b111d236","Type":"ContainerStarted","Data":"e1e1031986f0b0e9cbdf1c667869ad09e48f6a959b6f40bb2605de8c06fead77"}
Sep 10 14:32:01.064730 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:01.064693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n" event={"ID":"c8c37460-c57f-4fb2-801d-a559b111d236","Type":"ContainerStarted","Data":"d44be8f93c10512ad42749a49a58f3ab99152f3d4e5feaf73deff028a7736094"}
Sep 10 14:32:01.065136 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:01.064764    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:32:01.097286 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:01.097232    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n" podStartSLOduration=2.74584893 podStartE2EDuration="3.097213167s" totalImagesPullingTime="351.364237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:32:01.093258902 +0000 UTC m=+739.284787100" watchObservedRunningTime="2026-09-10 14:32:01.097213167 +0000 UTC m=+739.288741409"
Sep 10 14:32:07.075220 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:07.075190    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:32:18.390133 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:18.390096    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"]
Sep 10 14:32:18.390655 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:18.390306    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s" podUID="b6feac28-ede3-460c-83eb-b7622f8b562d" containerName="maas-api" containerID="cri-o://4662f61f545d710563265229d3eb9e0154e6aa70064d78e8171a0ea192274453" gracePeriod=30
Sep 10 14:32:18.638849 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:18.638820    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:32:18.742440 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:18.742346    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6feac28-ede3-460c-83eb-b7622f8b562d-metrics-certs\") pod \"b6feac28-ede3-460c-83eb-b7622f8b562d\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") "
Sep 10 14:32:18.742440 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:18.742393    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6feac28-ede3-460c-83eb-b7622f8b562d-maas-api-tls\") pod \"b6feac28-ede3-460c-83eb-b7622f8b562d\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") "
Sep 10 14:32:18.742440 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:18.742422    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6feac28-ede3-460c-83eb-b7622f8b562d-openshift-service-ca\") pod \"b6feac28-ede3-460c-83eb-b7622f8b562d\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") "
Sep 10 14:32:18.742716 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:18.742530    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmv8p\" (UniqueName: \"kubernetes.io/projected/b6feac28-ede3-460c-83eb-b7622f8b562d-kube-api-access-gmv8p\") pod \"b6feac28-ede3-460c-83eb-b7622f8b562d\" (UID: \"b6feac28-ede3-460c-83eb-b7622f8b562d\") "
Sep 10 14:32:18.742987 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:18.742791    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6feac28-ede3-460c-83eb-b7622f8b562d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b6feac28-ede3-460c-83eb-b7622f8b562d" (UID: "b6feac28-ede3-460c-83eb-b7622f8b562d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:32:18.742987 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:18.742945    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6feac28-ede3-460c-83eb-b7622f8b562d-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:32:18.745312 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:18.745246    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6feac28-ede3-460c-83eb-b7622f8b562d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b6feac28-ede3-460c-83eb-b7622f8b562d" (UID: "b6feac28-ede3-460c-83eb-b7622f8b562d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:32:18.745312 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:18.745266    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6feac28-ede3-460c-83eb-b7622f8b562d-kube-api-access-gmv8p" (OuterVolumeSpecName: "kube-api-access-gmv8p") pod "b6feac28-ede3-460c-83eb-b7622f8b562d" (UID: "b6feac28-ede3-460c-83eb-b7622f8b562d"). InnerVolumeSpecName "kube-api-access-gmv8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:32:18.745312 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:18.745296    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6feac28-ede3-460c-83eb-b7622f8b562d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b6feac28-ede3-460c-83eb-b7622f8b562d" (UID: "b6feac28-ede3-460c-83eb-b7622f8b562d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:32:18.843499 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:18.843465    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6feac28-ede3-460c-83eb-b7622f8b562d-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:32:18.843499 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:18.843496    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6feac28-ede3-460c-83eb-b7622f8b562d-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:32:18.843499 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:18.843505    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmv8p\" (UniqueName: \"kubernetes.io/projected/b6feac28-ede3-460c-83eb-b7622f8b562d-kube-api-access-gmv8p\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:32:19.175452 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:19.175410    2580 generic.go:356] "Generic (PLEG): container finished" podID="b6feac28-ede3-460c-83eb-b7622f8b562d" containerID="4662f61f545d710563265229d3eb9e0154e6aa70064d78e8171a0ea192274453" exitCode=0
Sep 10 14:32:19.175452 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:19.175456    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"
Sep 10 14:32:19.175654 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:19.175461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s" event={"ID":"b6feac28-ede3-460c-83eb-b7622f8b562d","Type":"ContainerDied","Data":"4662f61f545d710563265229d3eb9e0154e6aa70064d78e8171a0ea192274453"}
Sep 10 14:32:19.175654 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:19.175512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s" event={"ID":"b6feac28-ede3-460c-83eb-b7622f8b562d","Type":"ContainerDied","Data":"943238dd0b8c901565f8fe1a7843fd8f98c5e7e9c8f5dd4a9ccc5500bb1f5925"}
Sep 10 14:32:19.175654 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:19.175544    2580 scope.go:117] "RemoveContainer" containerID="4662f61f545d710563265229d3eb9e0154e6aa70064d78e8171a0ea192274453"
Sep 10 14:32:19.185810 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:19.185787    2580 scope.go:117] "RemoveContainer" containerID="4662f61f545d710563265229d3eb9e0154e6aa70064d78e8171a0ea192274453"
Sep 10 14:32:19.186096 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:32:19.186076    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4662f61f545d710563265229d3eb9e0154e6aa70064d78e8171a0ea192274453\": container with ID starting with 4662f61f545d710563265229d3eb9e0154e6aa70064d78e8171a0ea192274453 not found: ID does not exist" containerID="4662f61f545d710563265229d3eb9e0154e6aa70064d78e8171a0ea192274453"
Sep 10 14:32:19.186175 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:19.186111    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4662f61f545d710563265229d3eb9e0154e6aa70064d78e8171a0ea192274453"} err="failed to get container status \"4662f61f545d710563265229d3eb9e0154e6aa70064d78e8171a0ea192274453\": rpc error: code = NotFound desc = could not find container \"4662f61f545d710563265229d3eb9e0154e6aa70064d78e8171a0ea192274453\": container with ID starting with 4662f61f545d710563265229d3eb9e0154e6aa70064d78e8171a0ea192274453 not found: ID does not exist"
Sep 10 14:32:19.207509 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:19.207471    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"]
Sep 10 14:32:19.230473 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:19.230435    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad4f3b47-77754456c9-75v4s"]
Sep 10 14:32:20.310089 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:20.310058    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6feac28-ede3-460c-83eb-b7622f8b562d" path="/var/lib/kubelet/pods/b6feac28-ede3-460c-83eb-b7622f8b562d/volumes"
Sep 10 14:32:22.592094 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.592057    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"]
Sep 10 14:32:22.592706 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.592684    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6feac28-ede3-460c-83eb-b7622f8b562d" containerName="maas-api"
Sep 10 14:32:22.592706 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.592708    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6feac28-ede3-460c-83eb-b7622f8b562d" containerName="maas-api"
Sep 10 14:32:22.592859 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.592822    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6feac28-ede3-460c-83eb-b7622f8b562d" containerName="maas-api"
Sep 10 14:32:22.595600 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.595578    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:22.598160 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.598133    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d320da1d-serving-cert\""
Sep 10 14:32:22.605681 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.605657    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"]
Sep 10 14:32:22.683578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.683537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/928144a5-e0c8-47a2-ab02-27a80f424216-openshift-service-ca\") pod \"maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:22.683578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.683575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5sfv\" (UniqueName: \"kubernetes.io/projected/928144a5-e0c8-47a2-ab02-27a80f424216-kube-api-access-v5sfv\") pod \"maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:22.683844 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.683703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928144a5-e0c8-47a2-ab02-27a80f424216-maas-api-tls\") pod \"maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:22.683844 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.683743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928144a5-e0c8-47a2-ab02-27a80f424216-metrics-certs\") pod \"maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:22.784396 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.784356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928144a5-e0c8-47a2-ab02-27a80f424216-maas-api-tls\") pod \"maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:22.784396 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.784397    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928144a5-e0c8-47a2-ab02-27a80f424216-metrics-certs\") pod \"maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:22.784658 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.784483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/928144a5-e0c8-47a2-ab02-27a80f424216-openshift-service-ca\") pod \"maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:22.784658 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.784508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5sfv\" (UniqueName: \"kubernetes.io/projected/928144a5-e0c8-47a2-ab02-27a80f424216-kube-api-access-v5sfv\") pod \"maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:22.785169 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.785140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/928144a5-e0c8-47a2-ab02-27a80f424216-openshift-service-ca\") pod \"maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:22.786834 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.786812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928144a5-e0c8-47a2-ab02-27a80f424216-metrics-certs\") pod \"maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:22.786930 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.786876    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928144a5-e0c8-47a2-ab02-27a80f424216-maas-api-tls\") pod \"maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:22.793023 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.793004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5sfv\" (UniqueName: \"kubernetes.io/projected/928144a5-e0c8-47a2-ab02-27a80f424216-kube-api-access-v5sfv\") pod \"maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:22.908238 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:22.908202    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:23.186365 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:23.186339    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"]
Sep 10 14:32:23.188220 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:32:23.188178    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928144a5_e0c8_47a2_ab02_27a80f424216.slice/crio-8004968cfea0cd16b3c8ad0742dab9d0b69abd9ae0b5915b31529e8060025dc8 WatchSource:0}: Error finding container 8004968cfea0cd16b3c8ad0742dab9d0b69abd9ae0b5915b31529e8060025dc8: Status 404 returned error can't find the container with id 8004968cfea0cd16b3c8ad0742dab9d0b69abd9ae0b5915b31529e8060025dc8
Sep 10 14:32:23.200517 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:23.200490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr" event={"ID":"928144a5-e0c8-47a2-ab02-27a80f424216","Type":"ContainerStarted","Data":"8004968cfea0cd16b3c8ad0742dab9d0b69abd9ae0b5915b31529e8060025dc8"}
Sep 10 14:32:24.208200 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:24.208163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr" event={"ID":"928144a5-e0c8-47a2-ab02-27a80f424216","Type":"ContainerStarted","Data":"0b827d60b65c75a443b0a4bc0986af9595452206195fa94cb65d99ad0706ed35"}
Sep 10 14:32:24.208599 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:24.208263    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:24.273141 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:24.273084    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr" podStartSLOduration=1.6707587240000001 podStartE2EDuration="2.273067649s" totalImagesPullingTime="602.308925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:32:24.270505163 +0000 UTC m=+762.462033372" watchObservedRunningTime="2026-09-10 14:32:24.273067649 +0000 UTC m=+762.464595847"
Sep 10 14:32:30.219431 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:30.219403    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:53.925522 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:53.925489    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"]
Sep 10 14:32:53.926075 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:53.925754    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr" podUID="928144a5-e0c8-47a2-ab02-27a80f424216" containerName="maas-api" containerID="cri-o://0b827d60b65c75a443b0a4bc0986af9595452206195fa94cb65d99ad0706ed35" gracePeriod=30
Sep 10 14:32:54.177660 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.177598    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:54.188559 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.188535    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928144a5-e0c8-47a2-ab02-27a80f424216-metrics-certs\") pod \"928144a5-e0c8-47a2-ab02-27a80f424216\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") "
Sep 10 14:32:54.188671 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.188600    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/928144a5-e0c8-47a2-ab02-27a80f424216-openshift-service-ca\") pod \"928144a5-e0c8-47a2-ab02-27a80f424216\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") "
Sep 10 14:32:54.188671 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.188632    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5sfv\" (UniqueName: \"kubernetes.io/projected/928144a5-e0c8-47a2-ab02-27a80f424216-kube-api-access-v5sfv\") pod \"928144a5-e0c8-47a2-ab02-27a80f424216\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") "
Sep 10 14:32:54.188671 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.188654    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928144a5-e0c8-47a2-ab02-27a80f424216-maas-api-tls\") pod \"928144a5-e0c8-47a2-ab02-27a80f424216\" (UID: \"928144a5-e0c8-47a2-ab02-27a80f424216\") "
Sep 10 14:32:54.189003 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.188963    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/928144a5-e0c8-47a2-ab02-27a80f424216-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "928144a5-e0c8-47a2-ab02-27a80f424216" (UID: "928144a5-e0c8-47a2-ab02-27a80f424216"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:32:54.190751 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.190718    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/928144a5-e0c8-47a2-ab02-27a80f424216-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "928144a5-e0c8-47a2-ab02-27a80f424216" (UID: "928144a5-e0c8-47a2-ab02-27a80f424216"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:32:54.190871 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.190785    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928144a5-e0c8-47a2-ab02-27a80f424216-kube-api-access-v5sfv" (OuterVolumeSpecName: "kube-api-access-v5sfv") pod "928144a5-e0c8-47a2-ab02-27a80f424216" (UID: "928144a5-e0c8-47a2-ab02-27a80f424216"). InnerVolumeSpecName "kube-api-access-v5sfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:32:54.190871 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.190855    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/928144a5-e0c8-47a2-ab02-27a80f424216-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "928144a5-e0c8-47a2-ab02-27a80f424216" (UID: "928144a5-e0c8-47a2-ab02-27a80f424216"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:32:54.290042 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.290007    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928144a5-e0c8-47a2-ab02-27a80f424216-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:32:54.290042 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.290046    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/928144a5-e0c8-47a2-ab02-27a80f424216-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:32:54.290235 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.290062    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5sfv\" (UniqueName: \"kubernetes.io/projected/928144a5-e0c8-47a2-ab02-27a80f424216-kube-api-access-v5sfv\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:32:54.290235 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.290075    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928144a5-e0c8-47a2-ab02-27a80f424216-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:32:54.379875 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.379841    2580 generic.go:356] "Generic (PLEG): container finished" podID="928144a5-e0c8-47a2-ab02-27a80f424216" containerID="0b827d60b65c75a443b0a4bc0986af9595452206195fa94cb65d99ad0706ed35" exitCode=0
Sep 10 14:32:54.380098 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.379889    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"
Sep 10 14:32:54.380098 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.379909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr" event={"ID":"928144a5-e0c8-47a2-ab02-27a80f424216","Type":"ContainerDied","Data":"0b827d60b65c75a443b0a4bc0986af9595452206195fa94cb65d99ad0706ed35"}
Sep 10 14:32:54.380098 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.379945    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr" event={"ID":"928144a5-e0c8-47a2-ab02-27a80f424216","Type":"ContainerDied","Data":"8004968cfea0cd16b3c8ad0742dab9d0b69abd9ae0b5915b31529e8060025dc8"}
Sep 10 14:32:54.380098 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.379990    2580 scope.go:117] "RemoveContainer" containerID="0b827d60b65c75a443b0a4bc0986af9595452206195fa94cb65d99ad0706ed35"
Sep 10 14:32:54.389763 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.389745    2580 scope.go:117] "RemoveContainer" containerID="0b827d60b65c75a443b0a4bc0986af9595452206195fa94cb65d99ad0706ed35"
Sep 10 14:32:54.390100 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:32:54.390082    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b827d60b65c75a443b0a4bc0986af9595452206195fa94cb65d99ad0706ed35\": container with ID starting with 0b827d60b65c75a443b0a4bc0986af9595452206195fa94cb65d99ad0706ed35 not found: ID does not exist" containerID="0b827d60b65c75a443b0a4bc0986af9595452206195fa94cb65d99ad0706ed35"
Sep 10 14:32:54.390155 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.390111    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b827d60b65c75a443b0a4bc0986af9595452206195fa94cb65d99ad0706ed35"} err="failed to get container status \"0b827d60b65c75a443b0a4bc0986af9595452206195fa94cb65d99ad0706ed35\": rpc error: code = NotFound desc = could not find container \"0b827d60b65c75a443b0a4bc0986af9595452206195fa94cb65d99ad0706ed35\": container with ID starting with 0b827d60b65c75a443b0a4bc0986af9595452206195fa94cb65d99ad0706ed35 not found: ID does not exist"
Sep 10 14:32:54.397753 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.397727    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"]
Sep 10 14:32:54.410502 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:54.410468    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d320da1d-6989d54ccb-56kpr"]
Sep 10 14:32:56.315202 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:56.315166    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="928144a5-e0c8-47a2-ab02-27a80f424216" path="/var/lib/kubelet/pods/928144a5-e0c8-47a2-ab02-27a80f424216/volumes"
Sep 10 14:32:58.935985 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:58.935932    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"]
Sep 10 14:32:58.936354 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:58.936340    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="928144a5-e0c8-47a2-ab02-27a80f424216" containerName="maas-api"
Sep 10 14:32:58.936397 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:58.936355    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="928144a5-e0c8-47a2-ab02-27a80f424216" containerName="maas-api"
Sep 10 14:32:58.936443 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:58.936434    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="928144a5-e0c8-47a2-ab02-27a80f424216" containerName="maas-api"
Sep 10 14:32:58.939509 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:58.939486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:32:58.941820 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:58.941797    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-49ac64b7-serving-cert\""
Sep 10 14:32:58.951361 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:58.951319    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"]
Sep 10 14:32:59.035099 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:59.035060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9670ec35-cfaf-4f4c-95db-2ddb708888bf-metrics-certs\") pod \"maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:32:59.035304 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:59.035127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9670ec35-cfaf-4f4c-95db-2ddb708888bf-maas-api-tls\") pod \"maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:32:59.035304 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:59.035153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9670ec35-cfaf-4f4c-95db-2ddb708888bf-openshift-service-ca\") pod \"maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:32:59.035304 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:59.035230    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b6vb\" (UniqueName: \"kubernetes.io/projected/9670ec35-cfaf-4f4c-95db-2ddb708888bf-kube-api-access-5b6vb\") pod \"maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:32:59.136269 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:59.136230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9670ec35-cfaf-4f4c-95db-2ddb708888bf-metrics-certs\") pod \"maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:32:59.136464 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:59.136287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9670ec35-cfaf-4f4c-95db-2ddb708888bf-maas-api-tls\") pod \"maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:32:59.136464 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:59.136304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9670ec35-cfaf-4f4c-95db-2ddb708888bf-openshift-service-ca\") pod \"maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:32:59.136464 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:59.136355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b6vb\" (UniqueName: \"kubernetes.io/projected/9670ec35-cfaf-4f4c-95db-2ddb708888bf-kube-api-access-5b6vb\") pod \"maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:32:59.137047 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:59.137016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9670ec35-cfaf-4f4c-95db-2ddb708888bf-openshift-service-ca\") pod \"maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:32:59.138735 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:59.138711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9670ec35-cfaf-4f4c-95db-2ddb708888bf-maas-api-tls\") pod \"maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:32:59.138837 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:59.138737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9670ec35-cfaf-4f4c-95db-2ddb708888bf-metrics-certs\") pod \"maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:32:59.145030 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:59.145005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b6vb\" (UniqueName: \"kubernetes.io/projected/9670ec35-cfaf-4f4c-95db-2ddb708888bf-kube-api-access-5b6vb\") pod \"maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:32:59.254391 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:59.254290    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:32:59.530681 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:32:59.530655    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"]
Sep 10 14:32:59.532303 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:32:59.532275    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9670ec35_cfaf_4f4c_95db_2ddb708888bf.slice/crio-3a6f8ac48c0de7eab1e05ac7a719cd73e13f639fea54c5e71ec145b38b99d796 WatchSource:0}: Error finding container 3a6f8ac48c0de7eab1e05ac7a719cd73e13f639fea54c5e71ec145b38b99d796: Status 404 returned error can't find the container with id 3a6f8ac48c0de7eab1e05ac7a719cd73e13f639fea54c5e71ec145b38b99d796
Sep 10 14:33:00.416119 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:00.416074    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz" event={"ID":"9670ec35-cfaf-4f4c-95db-2ddb708888bf","Type":"ContainerStarted","Data":"3a6f8ac48c0de7eab1e05ac7a719cd73e13f639fea54c5e71ec145b38b99d796"}
Sep 10 14:33:01.422912 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:01.422879    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz" event={"ID":"9670ec35-cfaf-4f4c-95db-2ddb708888bf","Type":"ContainerStarted","Data":"194009d9da3c536e66622a298fa608ef45bc37f2bf67f57445b97fb558216513"}
Sep 10 14:33:01.423306 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:01.423024    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:33:01.442621 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:01.442563    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz" podStartSLOduration=2.510776335 podStartE2EDuration="3.44254909s" totalImagesPullingTime="931.772755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:33:01.439213307 +0000 UTC m=+799.630741530" watchObservedRunningTime="2026-09-10 14:33:01.44254909 +0000 UTC m=+799.634077289"
Sep 10 14:33:07.441999 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:07.441910    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:33:21.712656 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.712619    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"]
Sep 10 14:33:21.715278 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.715258    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.717795 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.717768    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-12068cef-kserve-self-signed-certs\""
Sep 10 14:33:21.724825 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.724801    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"]
Sep 10 14:33:21.849920 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.849883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-dshm\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.850098 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.849930    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-home\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.850098 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.850009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-tmp-dir\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.850098 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.850090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h52c\" (UniqueName: \"kubernetes.io/projected/c9b062b8-1899-4fd7-8857-f0d522e49025-kube-api-access-4h52c\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.850211 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.850117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-model-cache\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.850211 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.850161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9b062b8-1899-4fd7-8857-f0d522e49025-tls-certs\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.951592 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.951550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-tmp-dir\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.951771 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.951602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h52c\" (UniqueName: \"kubernetes.io/projected/c9b062b8-1899-4fd7-8857-f0d522e49025-kube-api-access-4h52c\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.951771 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.951726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-model-cache\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.951867 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.951807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9b062b8-1899-4fd7-8857-f0d522e49025-tls-certs\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.951932 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.951915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-dshm\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.952023 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.952003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-home\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.952093 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.952026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-tmp-dir\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.952093 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.952076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-model-cache\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.952328 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.952301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-home\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.954297 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.954251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-dshm\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.954406 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.954334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9b062b8-1899-4fd7-8857-f0d522e49025-tls-certs\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:21.959715 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:21.959692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h52c\" (UniqueName: \"kubernetes.io/projected/c9b062b8-1899-4fd7-8857-f0d522e49025-kube-api-access-4h52c\") pod \"e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") " pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:22.026598 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:22.026495    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:22.301515 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:22.301487    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"]
Sep 10 14:33:22.302839 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:33:22.302807    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9b062b8_1899_4fd7_8857_f0d522e49025.slice/crio-3e5bd8a4f00602365ecfe40ef07821b5e321a9d88995dfeaffce2046e4a79615 WatchSource:0}: Error finding container 3e5bd8a4f00602365ecfe40ef07821b5e321a9d88995dfeaffce2046e4a79615: Status 404 returned error can't find the container with id 3e5bd8a4f00602365ecfe40ef07821b5e321a9d88995dfeaffce2046e4a79615
Sep 10 14:33:22.546901 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:22.546869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl" event={"ID":"c9b062b8-1899-4fd7-8857-f0d522e49025","Type":"ContainerStarted","Data":"1e8a7c0cde8a1e9162e6648dea8bbc7f5bad8e75b5f193696204290ebc8101bc"}
Sep 10 14:33:22.547070 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:22.546912    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl" event={"ID":"c9b062b8-1899-4fd7-8857-f0d522e49025","Type":"ContainerStarted","Data":"3e5bd8a4f00602365ecfe40ef07821b5e321a9d88995dfeaffce2046e4a79615"}
Sep 10 14:33:22.564935 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:22.564835    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl" podStartSLOduration=1.564818783 podStartE2EDuration="1.564818783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:33:22.563894725 +0000 UTC m=+820.755422923" watchObservedRunningTime="2026-09-10 14:33:22.564818783 +0000 UTC m=+820.756347030"
Sep 10 14:33:29.432474 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.432430    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"]
Sep 10 14:33:29.432960 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.432806    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz" podUID="9670ec35-cfaf-4f4c-95db-2ddb708888bf" containerName="maas-api" containerID="cri-o://194009d9da3c536e66622a298fa608ef45bc37f2bf67f57445b97fb558216513" gracePeriod=30
Sep 10 14:33:29.589914 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.589879    2580 generic.go:356] "Generic (PLEG): container finished" podID="9670ec35-cfaf-4f4c-95db-2ddb708888bf" containerID="194009d9da3c536e66622a298fa608ef45bc37f2bf67f57445b97fb558216513" exitCode=0
Sep 10 14:33:29.590109 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.589964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz" event={"ID":"9670ec35-cfaf-4f4c-95db-2ddb708888bf","Type":"ContainerDied","Data":"194009d9da3c536e66622a298fa608ef45bc37f2bf67f57445b97fb558216513"}
Sep 10 14:33:29.685257 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.685190    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:33:29.822846 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.822810    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9670ec35-cfaf-4f4c-95db-2ddb708888bf-maas-api-tls\") pod \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") "
Sep 10 14:33:29.823059 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.822919    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9670ec35-cfaf-4f4c-95db-2ddb708888bf-openshift-service-ca\") pod \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") "
Sep 10 14:33:29.823059 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.822948    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9670ec35-cfaf-4f4c-95db-2ddb708888bf-metrics-certs\") pod \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") "
Sep 10 14:33:29.823059 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.823043    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b6vb\" (UniqueName: \"kubernetes.io/projected/9670ec35-cfaf-4f4c-95db-2ddb708888bf-kube-api-access-5b6vb\") pod \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\" (UID: \"9670ec35-cfaf-4f4c-95db-2ddb708888bf\") "
Sep 10 14:33:29.823370 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.823339    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9670ec35-cfaf-4f4c-95db-2ddb708888bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9670ec35-cfaf-4f4c-95db-2ddb708888bf" (UID: "9670ec35-cfaf-4f4c-95db-2ddb708888bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:33:29.825163 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.825136    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9670ec35-cfaf-4f4c-95db-2ddb708888bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9670ec35-cfaf-4f4c-95db-2ddb708888bf" (UID: "9670ec35-cfaf-4f4c-95db-2ddb708888bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:33:29.825417 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.825387    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9670ec35-cfaf-4f4c-95db-2ddb708888bf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9670ec35-cfaf-4f4c-95db-2ddb708888bf" (UID: "9670ec35-cfaf-4f4c-95db-2ddb708888bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:33:29.825515 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.825414    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9670ec35-cfaf-4f4c-95db-2ddb708888bf-kube-api-access-5b6vb" (OuterVolumeSpecName: "kube-api-access-5b6vb") pod "9670ec35-cfaf-4f4c-95db-2ddb708888bf" (UID: "9670ec35-cfaf-4f4c-95db-2ddb708888bf"). InnerVolumeSpecName "kube-api-access-5b6vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:33:29.924170 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.924128    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b6vb\" (UniqueName: \"kubernetes.io/projected/9670ec35-cfaf-4f4c-95db-2ddb708888bf-kube-api-access-5b6vb\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:33:29.924170 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.924163    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9670ec35-cfaf-4f4c-95db-2ddb708888bf-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:33:29.924170 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.924174    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9670ec35-cfaf-4f4c-95db-2ddb708888bf-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:33:29.924404 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:29.924183    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9670ec35-cfaf-4f4c-95db-2ddb708888bf-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:33:30.597229 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:30.597155    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"
Sep 10 14:33:30.597662 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:30.597146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz" event={"ID":"9670ec35-cfaf-4f4c-95db-2ddb708888bf","Type":"ContainerDied","Data":"3a6f8ac48c0de7eab1e05ac7a719cd73e13f639fea54c5e71ec145b38b99d796"}
Sep 10 14:33:30.597662 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:30.597296    2580 scope.go:117] "RemoveContainer" containerID="194009d9da3c536e66622a298fa608ef45bc37f2bf67f57445b97fb558216513"
Sep 10 14:33:30.617210 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:30.617182    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"]
Sep 10 14:33:30.620447 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:30.620424    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49ac64b7-5fc7999d68-xjklz"]
Sep 10 14:33:32.027168 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:32.027132    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:32.027630 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:32.027189    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:32.039879 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:32.039850    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:32.310136 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:32.310063    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9670ec35-cfaf-4f4c-95db-2ddb708888bf" path="/var/lib/kubelet/pods/9670ec35-cfaf-4f4c-95db-2ddb708888bf/volumes"
Sep 10 14:33:32.622451 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:32.622357    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:33:34.602792 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.602752    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"]
Sep 10 14:33:34.603208 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.603192    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9670ec35-cfaf-4f4c-95db-2ddb708888bf" containerName="maas-api"
Sep 10 14:33:34.603208 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.603209    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9670ec35-cfaf-4f4c-95db-2ddb708888bf" containerName="maas-api"
Sep 10 14:33:34.603296 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.603283    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9670ec35-cfaf-4f4c-95db-2ddb708888bf" containerName="maas-api"
Sep 10 14:33:34.605498 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.605482    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.608130 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.608104    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-d24d4c\"/\"openshift-service-ca.crt\""
Sep 10 14:33:34.609111 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.609091    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-d24d4c\"/\"kube-root-ca.crt\""
Sep 10 14:33:34.609222 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.609145    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-d24d4c\"/\"default-dockercfg-29wn6\""
Sep 10 14:33:34.609295 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.609221    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-d24d4c\"/\"autores-model-d24d4c-kserve-self-signed-certs\""
Sep 10 14:33:34.614719 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.614696    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"]
Sep 10 14:33:34.669169 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.669135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbdwn\" (UniqueName: \"kubernetes.io/projected/bbbce3b1-5815-4b80-ab29-48819a5c22ee-kube-api-access-bbdwn\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.669349 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.669183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbbce3b1-5815-4b80-ab29-48819a5c22ee-tls-certs\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.669349 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.669272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-home\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.669431 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.669378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-model-cache\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.669431 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.669405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-tmp-dir\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.669501 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.669435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-dshm\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.770259 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.770221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-model-cache\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.770259 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.770255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-tmp-dir\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.770492 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.770283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-dshm\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.770492 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.770313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbdwn\" (UniqueName: \"kubernetes.io/projected/bbbce3b1-5815-4b80-ab29-48819a5c22ee-kube-api-access-bbdwn\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.770492 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.770336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbbce3b1-5815-4b80-ab29-48819a5c22ee-tls-certs\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.770492 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.770359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-home\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.770704 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.770676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-tmp-dir\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.770765 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.770699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-model-cache\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.770765 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.770743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-home\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.772563 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.772540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-dshm\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.772781 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.772765    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbbce3b1-5815-4b80-ab29-48819a5c22ee-tls-certs\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.778278 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.778260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbdwn\" (UniqueName: \"kubernetes.io/projected/bbbce3b1-5815-4b80-ab29-48819a5c22ee-kube-api-access-bbdwn\") pod \"autores-model-d24d4c-kserve-6c6ffd869f-v9klq\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") " pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.798295 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.798262    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"]
Sep 10 14:33:34.809699 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.809671    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:34.813103 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.813063    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0245a1e4-serving-cert\""
Sep 10 14:33:34.819627 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.819593    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"]
Sep 10 14:33:34.870966 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.870850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52f437f8-8836-423f-aea7-2d58cba153ec-openshift-service-ca\") pod \"maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:34.870966 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.870918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52f437f8-8836-423f-aea7-2d58cba153ec-maas-api-tls\") pod \"maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:34.871429 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.871347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbbmv\" (UniqueName: \"kubernetes.io/projected/52f437f8-8836-423f-aea7-2d58cba153ec-kube-api-access-bbbmv\") pod \"maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:34.871555 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.871439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52f437f8-8836-423f-aea7-2d58cba153ec-metrics-certs\") pod \"maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:34.915938 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.915898    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:34.972867 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.972645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52f437f8-8836-423f-aea7-2d58cba153ec-openshift-service-ca\") pod \"maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:34.972867 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.972720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52f437f8-8836-423f-aea7-2d58cba153ec-maas-api-tls\") pod \"maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:34.972867 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.972803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbbmv\" (UniqueName: \"kubernetes.io/projected/52f437f8-8836-423f-aea7-2d58cba153ec-kube-api-access-bbbmv\") pod \"maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:34.972867 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.972839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52f437f8-8836-423f-aea7-2d58cba153ec-metrics-certs\") pod \"maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:34.973854 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.973623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52f437f8-8836-423f-aea7-2d58cba153ec-openshift-service-ca\") pod \"maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:34.976473 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.976410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52f437f8-8836-423f-aea7-2d58cba153ec-maas-api-tls\") pod \"maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:34.977383 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.977353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52f437f8-8836-423f-aea7-2d58cba153ec-metrics-certs\") pod \"maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:34.984659 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:34.984626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbbmv\" (UniqueName: \"kubernetes.io/projected/52f437f8-8836-423f-aea7-2d58cba153ec-kube-api-access-bbbmv\") pod \"maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:35.124733 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:35.124645    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:35.196709 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:35.196681    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"]
Sep 10 14:33:35.199221 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:33:35.199183    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbbce3b1_5815_4b80_ab29_48819a5c22ee.slice/crio-70b06b776ac401cd2cd2a410992503b2a682d752ff3df856edcbcb2baeb6e5fe WatchSource:0}: Error finding container 70b06b776ac401cd2cd2a410992503b2a682d752ff3df856edcbcb2baeb6e5fe: Status 404 returned error can't find the container with id 70b06b776ac401cd2cd2a410992503b2a682d752ff3df856edcbcb2baeb6e5fe
Sep 10 14:33:35.407313 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:35.407279    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"]
Sep 10 14:33:35.409006 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:33:35.408982    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f437f8_8836_423f_aea7_2d58cba153ec.slice/crio-2905857b42622fab58c35cf0be21e7a14bd39b35e197fde2cc1ae904ca77c41a WatchSource:0}: Error finding container 2905857b42622fab58c35cf0be21e7a14bd39b35e197fde2cc1ae904ca77c41a: Status 404 returned error can't find the container with id 2905857b42622fab58c35cf0be21e7a14bd39b35e197fde2cc1ae904ca77c41a
Sep 10 14:33:35.628050 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:35.628015    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8" event={"ID":"52f437f8-8836-423f-aea7-2d58cba153ec","Type":"ContainerStarted","Data":"2905857b42622fab58c35cf0be21e7a14bd39b35e197fde2cc1ae904ca77c41a"}
Sep 10 14:33:35.629981 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:35.629941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq" event={"ID":"bbbce3b1-5815-4b80-ab29-48819a5c22ee","Type":"ContainerStarted","Data":"0689936929b002a91802cf54dbf459b585c003098eadcbff3880d95421a8e805"}
Sep 10 14:33:35.630116 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:35.629999    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq" event={"ID":"bbbce3b1-5815-4b80-ab29-48819a5c22ee","Type":"ContainerStarted","Data":"70b06b776ac401cd2cd2a410992503b2a682d752ff3df856edcbcb2baeb6e5fe"}
Sep 10 14:33:35.652478 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:35.652425    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq" podStartSLOduration=1.652410237 podStartE2EDuration="1.652410237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:33:35.651306159 +0000 UTC m=+833.842834356" watchObservedRunningTime="2026-09-10 14:33:35.652410237 +0000 UTC m=+833.843938474"
Sep 10 14:33:36.636773 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:36.636735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8" event={"ID":"52f437f8-8836-423f-aea7-2d58cba153ec","Type":"ContainerStarted","Data":"925bc1b69b232201e5d0a06f9f187dc6fd73d670e007f462023e210223586cb4"}
Sep 10 14:33:36.637264 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:36.636808    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:36.655650 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:36.655588    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8" podStartSLOduration=2.20993475 podStartE2EDuration="2.655568098s" totalImagesPullingTime="445.633348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:33:36.655125854 +0000 UTC m=+834.846654086" watchObservedRunningTime="2026-09-10 14:33:36.655568098 +0000 UTC m=+834.847096299"
Sep 10 14:33:42.649305 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:42.649275    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:33:44.916855 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:44.916808    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:44.916855 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:44.916866    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:44.929711 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:44.929685    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:45.705267 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:45.705237    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:33:56.799959 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:56.799923    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"]
Sep 10 14:33:57.520264 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:33:57.520228    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"]
Sep 10 14:34:05.563696 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:05.563661    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"]
Sep 10 14:34:05.564136 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:05.563862    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8" podUID="52f437f8-8836-423f-aea7-2d58cba153ec" containerName="maas-api" containerID="cri-o://925bc1b69b232201e5d0a06f9f187dc6fd73d670e007f462023e210223586cb4" gracePeriod=30
Sep 10 14:34:05.810103 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:05.810069    2580 generic.go:356] "Generic (PLEG): container finished" podID="52f437f8-8836-423f-aea7-2d58cba153ec" containerID="925bc1b69b232201e5d0a06f9f187dc6fd73d670e007f462023e210223586cb4" exitCode=0
Sep 10 14:34:05.810249 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:05.810103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8" event={"ID":"52f437f8-8836-423f-aea7-2d58cba153ec","Type":"ContainerDied","Data":"925bc1b69b232201e5d0a06f9f187dc6fd73d670e007f462023e210223586cb4"}
Sep 10 14:34:05.810249 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:05.810150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8" event={"ID":"52f437f8-8836-423f-aea7-2d58cba153ec","Type":"ContainerDied","Data":"2905857b42622fab58c35cf0be21e7a14bd39b35e197fde2cc1ae904ca77c41a"}
Sep 10 14:34:05.810249 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:05.810167    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2905857b42622fab58c35cf0be21e7a14bd39b35e197fde2cc1ae904ca77c41a"
Sep 10 14:34:05.816061 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:05.816010    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:34:05.964914 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:05.964859    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52f437f8-8836-423f-aea7-2d58cba153ec-maas-api-tls\") pod \"52f437f8-8836-423f-aea7-2d58cba153ec\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") "
Sep 10 14:34:05.965147 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:05.965001    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52f437f8-8836-423f-aea7-2d58cba153ec-openshift-service-ca\") pod \"52f437f8-8836-423f-aea7-2d58cba153ec\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") "
Sep 10 14:34:05.965147 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:05.965037    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbbmv\" (UniqueName: \"kubernetes.io/projected/52f437f8-8836-423f-aea7-2d58cba153ec-kube-api-access-bbbmv\") pod \"52f437f8-8836-423f-aea7-2d58cba153ec\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") "
Sep 10 14:34:05.965147 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:05.965065    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52f437f8-8836-423f-aea7-2d58cba153ec-metrics-certs\") pod \"52f437f8-8836-423f-aea7-2d58cba153ec\" (UID: \"52f437f8-8836-423f-aea7-2d58cba153ec\") "
Sep 10 14:34:05.965402 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:05.965370    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f437f8-8836-423f-aea7-2d58cba153ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52f437f8-8836-423f-aea7-2d58cba153ec" (UID: "52f437f8-8836-423f-aea7-2d58cba153ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:34:05.967085 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:05.967052    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f437f8-8836-423f-aea7-2d58cba153ec-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "52f437f8-8836-423f-aea7-2d58cba153ec" (UID: "52f437f8-8836-423f-aea7-2d58cba153ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:34:05.967194 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:05.967100    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f437f8-8836-423f-aea7-2d58cba153ec-kube-api-access-bbbmv" (OuterVolumeSpecName: "kube-api-access-bbbmv") pod "52f437f8-8836-423f-aea7-2d58cba153ec" (UID: "52f437f8-8836-423f-aea7-2d58cba153ec"). InnerVolumeSpecName "kube-api-access-bbbmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:34:05.967194 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:05.967132    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f437f8-8836-423f-aea7-2d58cba153ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52f437f8-8836-423f-aea7-2d58cba153ec" (UID: "52f437f8-8836-423f-aea7-2d58cba153ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:34:06.065908 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:06.065872    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52f437f8-8836-423f-aea7-2d58cba153ec-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:06.065908 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:06.065899    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52f437f8-8836-423f-aea7-2d58cba153ec-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:06.065908 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:06.065911    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbbmv\" (UniqueName: \"kubernetes.io/projected/52f437f8-8836-423f-aea7-2d58cba153ec-kube-api-access-bbbmv\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:06.066178 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:06.065920    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52f437f8-8836-423f-aea7-2d58cba153ec-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:06.815237 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:06.815204    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"
Sep 10 14:34:06.838928 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:06.838898    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"]
Sep 10 14:34:06.844856 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:06.844830    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0245a1e4-c9c684599-7m2t8"]
Sep 10 14:34:08.310617 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.310587    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52f437f8-8836-423f-aea7-2d58cba153ec" path="/var/lib/kubelet/pods/52f437f8-8836-423f-aea7-2d58cba153ec/volumes"
Sep 10 14:34:08.408152 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.408113    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"]
Sep 10 14:34:08.408420 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.408390    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z" podUID="2e1bc717-9c1a-4e73-a6f0-246bbb5994b4" containerName="maas-api" containerID="cri-o://b916b2155e2cec042af277eb4123305b062c1d0769df2e4b32a50afca39fa1b0" gracePeriod=30
Sep 10 14:34:08.476988 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.476945    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"]
Sep 10 14:34:08.477251 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.477222    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq" podUID="fb4c43af-fdbb-44f3-9ae2-e495bc5842ed" containerName="payload-pre-processing" containerID="cri-o://e437b2475bd9c7fe2fbb2b3d1d671c30003a880ad82aaffc487bd50a82911222" gracePeriod=30
Sep 10 14:34:08.718647 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.718621    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:34:08.756679 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.756650    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"
Sep 10 14:34:08.828789 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.828752    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb4c43af-fdbb-44f3-9ae2-e495bc5842ed" containerID="e437b2475bd9c7fe2fbb2b3d1d671c30003a880ad82aaffc487bd50a82911222" exitCode=0
Sep 10 14:34:08.829001 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.828803    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"
Sep 10 14:34:08.829001 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.828835    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq" event={"ID":"fb4c43af-fdbb-44f3-9ae2-e495bc5842ed","Type":"ContainerDied","Data":"e437b2475bd9c7fe2fbb2b3d1d671c30003a880ad82aaffc487bd50a82911222"}
Sep 10 14:34:08.829001 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.828914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq" event={"ID":"fb4c43af-fdbb-44f3-9ae2-e495bc5842ed","Type":"ContainerDied","Data":"84c6bfe26e8d3eb736e022c07e4fc5390c0f5d1ae0e62996b9da40fef698ba9f"}
Sep 10 14:34:08.829001 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.828938    2580 scope.go:117] "RemoveContainer" containerID="e437b2475bd9c7fe2fbb2b3d1d671c30003a880ad82aaffc487bd50a82911222"
Sep 10 14:34:08.830287 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.830256    2580 generic.go:356] "Generic (PLEG): container finished" podID="2e1bc717-9c1a-4e73-a6f0-246bbb5994b4" containerID="b916b2155e2cec042af277eb4123305b062c1d0769df2e4b32a50afca39fa1b0" exitCode=0
Sep 10 14:34:08.830375 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.830311    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"
Sep 10 14:34:08.830375 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.830326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z" event={"ID":"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4","Type":"ContainerDied","Data":"b916b2155e2cec042af277eb4123305b062c1d0769df2e4b32a50afca39fa1b0"}
Sep 10 14:34:08.830375 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.830366    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z" event={"ID":"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4","Type":"ContainerDied","Data":"cfcc08f366539ef4c9dbe8e9f5428b7f73803fa05d661e0507ad3382bac730ab"}
Sep 10 14:34:08.838271 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.838250    2580 scope.go:117] "RemoveContainer" containerID="e437b2475bd9c7fe2fbb2b3d1d671c30003a880ad82aaffc487bd50a82911222"
Sep 10 14:34:08.838556 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:34:08.838537    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e437b2475bd9c7fe2fbb2b3d1d671c30003a880ad82aaffc487bd50a82911222\": container with ID starting with e437b2475bd9c7fe2fbb2b3d1d671c30003a880ad82aaffc487bd50a82911222 not found: ID does not exist" containerID="e437b2475bd9c7fe2fbb2b3d1d671c30003a880ad82aaffc487bd50a82911222"
Sep 10 14:34:08.838600 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.838570    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e437b2475bd9c7fe2fbb2b3d1d671c30003a880ad82aaffc487bd50a82911222"} err="failed to get container status \"e437b2475bd9c7fe2fbb2b3d1d671c30003a880ad82aaffc487bd50a82911222\": rpc error: code = NotFound desc = could not find container \"e437b2475bd9c7fe2fbb2b3d1d671c30003a880ad82aaffc487bd50a82911222\": container with ID starting with e437b2475bd9c7fe2fbb2b3d1d671c30003a880ad82aaffc487bd50a82911222 not found: ID does not exist"
Sep 10 14:34:08.838600 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.838588    2580 scope.go:117] "RemoveContainer" containerID="b916b2155e2cec042af277eb4123305b062c1d0769df2e4b32a50afca39fa1b0"
Sep 10 14:34:08.847237 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.847217    2580 scope.go:117] "RemoveContainer" containerID="b916b2155e2cec042af277eb4123305b062c1d0769df2e4b32a50afca39fa1b0"
Sep 10 14:34:08.847508 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:34:08.847491    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b916b2155e2cec042af277eb4123305b062c1d0769df2e4b32a50afca39fa1b0\": container with ID starting with b916b2155e2cec042af277eb4123305b062c1d0769df2e4b32a50afca39fa1b0 not found: ID does not exist" containerID="b916b2155e2cec042af277eb4123305b062c1d0769df2e4b32a50afca39fa1b0"
Sep 10 14:34:08.847553 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.847518    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b916b2155e2cec042af277eb4123305b062c1d0769df2e4b32a50afca39fa1b0"} err="failed to get container status \"b916b2155e2cec042af277eb4123305b062c1d0769df2e4b32a50afca39fa1b0\": rpc error: code = NotFound desc = could not find container \"b916b2155e2cec042af277eb4123305b062c1d0769df2e4b32a50afca39fa1b0\": container with ID starting with b916b2155e2cec042af277eb4123305b062c1d0769df2e4b32a50afca39fa1b0 not found: ID does not exist"
Sep 10 14:34:08.892578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.892536    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb4c43af-fdbb-44f3-9ae2-e495bc5842ed-plugins-config-volume\") pod \"fb4c43af-fdbb-44f3-9ae2-e495bc5842ed\" (UID: \"fb4c43af-fdbb-44f3-9ae2-e495bc5842ed\") "
Sep 10 14:34:08.892711 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.892601    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kz4j\" (UniqueName: \"kubernetes.io/projected/fb4c43af-fdbb-44f3-9ae2-e495bc5842ed-kube-api-access-5kz4j\") pod \"fb4c43af-fdbb-44f3-9ae2-e495bc5842ed\" (UID: \"fb4c43af-fdbb-44f3-9ae2-e495bc5842ed\") "
Sep 10 14:34:08.892711 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.892641    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-maas-api-tls\") pod \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") "
Sep 10 14:34:08.892711 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.892674    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-metrics-certs\") pod \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") "
Sep 10 14:34:08.892859 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.892719    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-openshift-service-ca\") pod \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") "
Sep 10 14:34:08.892859 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.892769    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcxp4\" (UniqueName: \"kubernetes.io/projected/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-kube-api-access-rcxp4\") pod \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\" (UID: \"2e1bc717-9c1a-4e73-a6f0-246bbb5994b4\") "
Sep 10 14:34:08.893094 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.893052    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4c43af-fdbb-44f3-9ae2-e495bc5842ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb4c43af-fdbb-44f3-9ae2-e495bc5842ed" (UID: "fb4c43af-fdbb-44f3-9ae2-e495bc5842ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:34:08.893281 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.893184    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e1bc717-9c1a-4e73-a6f0-246bbb5994b4" (UID: "2e1bc717-9c1a-4e73-a6f0-246bbb5994b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:34:08.894824 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.894798    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2e1bc717-9c1a-4e73-a6f0-246bbb5994b4" (UID: "2e1bc717-9c1a-4e73-a6f0-246bbb5994b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:34:08.894921 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.894835    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e1bc717-9c1a-4e73-a6f0-246bbb5994b4" (UID: "2e1bc717-9c1a-4e73-a6f0-246bbb5994b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:34:08.895199 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.895176    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-kube-api-access-rcxp4" (OuterVolumeSpecName: "kube-api-access-rcxp4") pod "2e1bc717-9c1a-4e73-a6f0-246bbb5994b4" (UID: "2e1bc717-9c1a-4e73-a6f0-246bbb5994b4"). InnerVolumeSpecName "kube-api-access-rcxp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:34:08.895245 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.895228    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4c43af-fdbb-44f3-9ae2-e495bc5842ed-kube-api-access-5kz4j" (OuterVolumeSpecName: "kube-api-access-5kz4j") pod "fb4c43af-fdbb-44f3-9ae2-e495bc5842ed" (UID: "fb4c43af-fdbb-44f3-9ae2-e495bc5842ed"). InnerVolumeSpecName "kube-api-access-5kz4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:34:08.994478 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.994423    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb4c43af-fdbb-44f3-9ae2-e495bc5842ed-plugins-config-volume\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:08.994478 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.994470    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kz4j\" (UniqueName: \"kubernetes.io/projected/fb4c43af-fdbb-44f3-9ae2-e495bc5842ed-kube-api-access-5kz4j\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:08.994478 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.994481    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:08.994478 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.994491    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:08.994478 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.994499    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:08.994795 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:08.994508    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcxp4\" (UniqueName: \"kubernetes.io/projected/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4-kube-api-access-rcxp4\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:09.155285 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:09.155254    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"]
Sep 10 14:34:09.157478 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:09.157456    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d24d4c-79ccc9d5fd-nhd2z"]
Sep 10 14:34:09.167088 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:09.167062    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"]
Sep 10 14:34:09.173184 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:09.173161    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d24d4c-7f984fcdd6-64qbq"]
Sep 10 14:34:10.310750 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:10.310719    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e1bc717-9c1a-4e73-a6f0-246bbb5994b4" path="/var/lib/kubelet/pods/2e1bc717-9c1a-4e73-a6f0-246bbb5994b4/volumes"
Sep 10 14:34:10.311137 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:10.311100    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb4c43af-fdbb-44f3-9ae2-e495bc5842ed" path="/var/lib/kubelet/pods/fb4c43af-fdbb-44f3-9ae2-e495bc5842ed/volumes"
Sep 10 14:34:11.820765 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:11.820701    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl" podUID="c9b062b8-1899-4fd7-8857-f0d522e49025" containerName="main" containerID="cri-o://1e8a7c0cde8a1e9162e6648dea8bbc7f5bad8e75b5f193696204290ebc8101bc" gracePeriod=45
Sep 10 14:34:12.075164 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.075100    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:34:12.224580 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.224538    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9b062b8-1899-4fd7-8857-f0d522e49025-tls-certs\") pod \"c9b062b8-1899-4fd7-8857-f0d522e49025\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") "
Sep 10 14:34:12.224741 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.224612    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-dshm\") pod \"c9b062b8-1899-4fd7-8857-f0d522e49025\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") "
Sep 10 14:34:12.224741 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.224726    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-home\") pod \"c9b062b8-1899-4fd7-8857-f0d522e49025\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") "
Sep 10 14:34:12.224868 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.224776    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h52c\" (UniqueName: \"kubernetes.io/projected/c9b062b8-1899-4fd7-8857-f0d522e49025-kube-api-access-4h52c\") pod \"c9b062b8-1899-4fd7-8857-f0d522e49025\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") "
Sep 10 14:34:12.224868 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.224826    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-model-cache\") pod \"c9b062b8-1899-4fd7-8857-f0d522e49025\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") "
Sep 10 14:34:12.224868 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.224859    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-tmp-dir\") pod \"c9b062b8-1899-4fd7-8857-f0d522e49025\" (UID: \"c9b062b8-1899-4fd7-8857-f0d522e49025\") "
Sep 10 14:34:12.225055 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.224958    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-home" (OuterVolumeSpecName: "home") pod "c9b062b8-1899-4fd7-8857-f0d522e49025" (UID: "c9b062b8-1899-4fd7-8857-f0d522e49025"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:34:12.225193 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.225165    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9b062b8-1899-4fd7-8857-f0d522e49025" (UID: "c9b062b8-1899-4fd7-8857-f0d522e49025"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:34:12.225310 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.225208    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9b062b8-1899-4fd7-8857-f0d522e49025" (UID: "c9b062b8-1899-4fd7-8857-f0d522e49025"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:34:12.225310 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.225302    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:12.225429 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.225317    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:12.225429 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.225330    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:12.227137 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.227112    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-dshm" (OuterVolumeSpecName: "dshm") pod "c9b062b8-1899-4fd7-8857-f0d522e49025" (UID: "c9b062b8-1899-4fd7-8857-f0d522e49025"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:34:12.227238 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.227135    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b062b8-1899-4fd7-8857-f0d522e49025-kube-api-access-4h52c" (OuterVolumeSpecName: "kube-api-access-4h52c") pod "c9b062b8-1899-4fd7-8857-f0d522e49025" (UID: "c9b062b8-1899-4fd7-8857-f0d522e49025"). InnerVolumeSpecName "kube-api-access-4h52c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:34:12.227238 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.227209    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9b062b8-1899-4fd7-8857-f0d522e49025-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9b062b8-1899-4fd7-8857-f0d522e49025" (UID: "c9b062b8-1899-4fd7-8857-f0d522e49025"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:34:12.326408 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.326336    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h52c\" (UniqueName: \"kubernetes.io/projected/c9b062b8-1899-4fd7-8857-f0d522e49025-kube-api-access-4h52c\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:12.326408 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.326365    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9b062b8-1899-4fd7-8857-f0d522e49025-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:12.326408 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.326378    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9b062b8-1899-4fd7-8857-f0d522e49025-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:12.540058 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.540018    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq" podUID="bbbce3b1-5815-4b80-ab29-48819a5c22ee" containerName="main" containerID="cri-o://0689936929b002a91802cf54dbf459b585c003098eadcbff3880d95421a8e805" gracePeriod=45
Sep 10 14:34:12.766985 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.766953    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:34:12.860549 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.860515    2580 generic.go:356] "Generic (PLEG): container finished" podID="c9b062b8-1899-4fd7-8857-f0d522e49025" containerID="1e8a7c0cde8a1e9162e6648dea8bbc7f5bad8e75b5f193696204290ebc8101bc" exitCode=0
Sep 10 14:34:12.861004 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.860580    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"
Sep 10 14:34:12.861004 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.860605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl" event={"ID":"c9b062b8-1899-4fd7-8857-f0d522e49025","Type":"ContainerDied","Data":"1e8a7c0cde8a1e9162e6648dea8bbc7f5bad8e75b5f193696204290ebc8101bc"}
Sep 10 14:34:12.861004 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.860652    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl" event={"ID":"c9b062b8-1899-4fd7-8857-f0d522e49025","Type":"ContainerDied","Data":"3e5bd8a4f00602365ecfe40ef07821b5e321a9d88995dfeaffce2046e4a79615"}
Sep 10 14:34:12.861004 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.860673    2580 scope.go:117] "RemoveContainer" containerID="1e8a7c0cde8a1e9162e6648dea8bbc7f5bad8e75b5f193696204290ebc8101bc"
Sep 10 14:34:12.862304 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.862218    2580 generic.go:356] "Generic (PLEG): container finished" podID="bbbce3b1-5815-4b80-ab29-48819a5c22ee" containerID="0689936929b002a91802cf54dbf459b585c003098eadcbff3880d95421a8e805" exitCode=0
Sep 10 14:34:12.862304 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.862248    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq" event={"ID":"bbbce3b1-5815-4b80-ab29-48819a5c22ee","Type":"ContainerDied","Data":"0689936929b002a91802cf54dbf459b585c003098eadcbff3880d95421a8e805"}
Sep 10 14:34:12.862304 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.862294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq" event={"ID":"bbbce3b1-5815-4b80-ab29-48819a5c22ee","Type":"ContainerDied","Data":"70b06b776ac401cd2cd2a410992503b2a682d752ff3df856edcbcb2baeb6e5fe"}
Sep 10 14:34:12.862503 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.862258    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"
Sep 10 14:34:12.869775 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.869755    2580 scope.go:117] "RemoveContainer" containerID="1e8a7c0cde8a1e9162e6648dea8bbc7f5bad8e75b5f193696204290ebc8101bc"
Sep 10 14:34:12.870089 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:34:12.870066    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8a7c0cde8a1e9162e6648dea8bbc7f5bad8e75b5f193696204290ebc8101bc\": container with ID starting with 1e8a7c0cde8a1e9162e6648dea8bbc7f5bad8e75b5f193696204290ebc8101bc not found: ID does not exist" containerID="1e8a7c0cde8a1e9162e6648dea8bbc7f5bad8e75b5f193696204290ebc8101bc"
Sep 10 14:34:12.870171 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.870098    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8a7c0cde8a1e9162e6648dea8bbc7f5bad8e75b5f193696204290ebc8101bc"} err="failed to get container status \"1e8a7c0cde8a1e9162e6648dea8bbc7f5bad8e75b5f193696204290ebc8101bc\": rpc error: code = NotFound desc = could not find container \"1e8a7c0cde8a1e9162e6648dea8bbc7f5bad8e75b5f193696204290ebc8101bc\": container with ID starting with 1e8a7c0cde8a1e9162e6648dea8bbc7f5bad8e75b5f193696204290ebc8101bc not found: ID does not exist"
Sep 10 14:34:12.870171 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.870116    2580 scope.go:117] "RemoveContainer" containerID="0689936929b002a91802cf54dbf459b585c003098eadcbff3880d95421a8e805"
Sep 10 14:34:12.878029 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.877999    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"]
Sep 10 14:34:12.878675 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.878661    2580 scope.go:117] "RemoveContainer" containerID="0689936929b002a91802cf54dbf459b585c003098eadcbff3880d95421a8e805"
Sep 10 14:34:12.878959 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:34:12.878939    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0689936929b002a91802cf54dbf459b585c003098eadcbff3880d95421a8e805\": container with ID starting with 0689936929b002a91802cf54dbf459b585c003098eadcbff3880d95421a8e805 not found: ID does not exist" containerID="0689936929b002a91802cf54dbf459b585c003098eadcbff3880d95421a8e805"
Sep 10 14:34:12.879044 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.878983    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0689936929b002a91802cf54dbf459b585c003098eadcbff3880d95421a8e805"} err="failed to get container status \"0689936929b002a91802cf54dbf459b585c003098eadcbff3880d95421a8e805\": rpc error: code = NotFound desc = could not find container \"0689936929b002a91802cf54dbf459b585c003098eadcbff3880d95421a8e805\": container with ID starting with 0689936929b002a91802cf54dbf459b585c003098eadcbff3880d95421a8e805 not found: ID does not exist"
Sep 10 14:34:12.882734 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.882710    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-12068cef-kserve-d57577f7d-dwckl"]
Sep 10 14:34:12.932072 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.931988    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-model-cache\") pod \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") "
Sep 10 14:34:12.932072 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.932041    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-tmp-dir\") pod \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") "
Sep 10 14:34:12.932072 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.932073    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbbce3b1-5815-4b80-ab29-48819a5c22ee-tls-certs\") pod \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") "
Sep 10 14:34:12.932350 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.932141    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-home\") pod \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") "
Sep 10 14:34:12.932350 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.932167    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbdwn\" (UniqueName: \"kubernetes.io/projected/bbbce3b1-5815-4b80-ab29-48819a5c22ee-kube-api-access-bbdwn\") pod \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") "
Sep 10 14:34:12.932350 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.932205    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-dshm\") pod \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\" (UID: \"bbbce3b1-5815-4b80-ab29-48819a5c22ee\") "
Sep 10 14:34:12.932350 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.932259    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-model-cache" (OuterVolumeSpecName: "model-cache") pod "bbbce3b1-5815-4b80-ab29-48819a5c22ee" (UID: "bbbce3b1-5815-4b80-ab29-48819a5c22ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:34:12.932350 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.932332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bbbce3b1-5815-4b80-ab29-48819a5c22ee" (UID: "bbbce3b1-5815-4b80-ab29-48819a5c22ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:34:12.932561 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.932433    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-home" (OuterVolumeSpecName: "home") pod "bbbce3b1-5815-4b80-ab29-48819a5c22ee" (UID: "bbbce3b1-5815-4b80-ab29-48819a5c22ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:34:12.932628 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.932570    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:12.932628 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.932584    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:12.932628 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.932594    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:12.934422 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.934383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbbce3b1-5815-4b80-ab29-48819a5c22ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bbbce3b1-5815-4b80-ab29-48819a5c22ee" (UID: "bbbce3b1-5815-4b80-ab29-48819a5c22ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:34:12.934538 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.934448    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-dshm" (OuterVolumeSpecName: "dshm") pod "bbbce3b1-5815-4b80-ab29-48819a5c22ee" (UID: "bbbce3b1-5815-4b80-ab29-48819a5c22ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:34:12.934538 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:12.934464    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbce3b1-5815-4b80-ab29-48819a5c22ee-kube-api-access-bbdwn" (OuterVolumeSpecName: "kube-api-access-bbdwn") pod "bbbce3b1-5815-4b80-ab29-48819a5c22ee" (UID: "bbbce3b1-5815-4b80-ab29-48819a5c22ee"). InnerVolumeSpecName "kube-api-access-bbdwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:34:13.033947 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:13.033891    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbbce3b1-5815-4b80-ab29-48819a5c22ee-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:13.033947 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:13.033938    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbbce3b1-5815-4b80-ab29-48819a5c22ee-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:13.033947 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:13.033950    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbdwn\" (UniqueName: \"kubernetes.io/projected/bbbce3b1-5815-4b80-ab29-48819a5c22ee-kube-api-access-bbdwn\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:34:13.185152 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:13.185078    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"]
Sep 10 14:34:13.188449 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:13.188424    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-d24d4c/autores-model-d24d4c-kserve-6c6ffd869f-v9klq"]
Sep 10 14:34:14.310565 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:14.310523    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbbce3b1-5815-4b80-ab29-48819a5c22ee" path="/var/lib/kubelet/pods/bbbce3b1-5815-4b80-ab29-48819a5c22ee/volumes"
Sep 10 14:34:14.311155 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:14.311134    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9b062b8-1899-4fd7-8857-f0d522e49025" path="/var/lib/kubelet/pods/c9b062b8-1899-4fd7-8857-f0d522e49025/volumes"
Sep 10 14:34:20.742284 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.742250    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"]
Sep 10 14:34:20.742812 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.742792    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbbce3b1-5815-4b80-ab29-48819a5c22ee" containerName="main"
Sep 10 14:34:20.742920 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.742815    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbbce3b1-5815-4b80-ab29-48819a5c22ee" containerName="main"
Sep 10 14:34:20.742920 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.742857    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4c43af-fdbb-44f3-9ae2-e495bc5842ed" containerName="payload-pre-processing"
Sep 10 14:34:20.742920 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.742868    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4c43af-fdbb-44f3-9ae2-e495bc5842ed" containerName="payload-pre-processing"
Sep 10 14:34:20.742920 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.742883    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e1bc717-9c1a-4e73-a6f0-246bbb5994b4" containerName="maas-api"
Sep 10 14:34:20.742920 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.742890    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e1bc717-9c1a-4e73-a6f0-246bbb5994b4" containerName="maas-api"
Sep 10 14:34:20.742920 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.742902    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52f437f8-8836-423f-aea7-2d58cba153ec" containerName="maas-api"
Sep 10 14:34:20.742920 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.742910    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="52f437f8-8836-423f-aea7-2d58cba153ec" containerName="maas-api"
Sep 10 14:34:20.743285 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.742930    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b062b8-1899-4fd7-8857-f0d522e49025" containerName="main"
Sep 10 14:34:20.743285 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.742936    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b062b8-1899-4fd7-8857-f0d522e49025" containerName="main"
Sep 10 14:34:20.743285 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.743035    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4c43af-fdbb-44f3-9ae2-e495bc5842ed" containerName="payload-pre-processing"
Sep 10 14:34:20.743285 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.743048    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b062b8-1899-4fd7-8857-f0d522e49025" containerName="main"
Sep 10 14:34:20.743285 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.743058    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e1bc717-9c1a-4e73-a6f0-246bbb5994b4" containerName="maas-api"
Sep 10 14:34:20.743285 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.743090    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="52f437f8-8836-423f-aea7-2d58cba153ec" containerName="maas-api"
Sep 10 14:34:20.743285 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.743098    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbbce3b1-5815-4b80-ab29-48819a5c22ee" containerName="main"
Sep 10 14:34:20.747640 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.747619    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:20.750526 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.750506    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-345a2c-serving-cert\""
Sep 10 14:34:20.762023 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.761947    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"]
Sep 10 14:34:20.805388 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.805356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb89bc9-7da7-4f10-b549-7e7fcb829515-openshift-service-ca\") pod \"maas-api-e2e-explicit-345a2c-7989bf656d-26nx8\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:20.805568 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.805403    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb89bc9-7da7-4f10-b549-7e7fcb829515-maas-api-tls\") pod \"maas-api-e2e-explicit-345a2c-7989bf656d-26nx8\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:20.805568 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.805492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eb89bc9-7da7-4f10-b549-7e7fcb829515-metrics-certs\") pod \"maas-api-e2e-explicit-345a2c-7989bf656d-26nx8\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:20.805568 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.805552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gkvq\" (UniqueName: \"kubernetes.io/projected/4eb89bc9-7da7-4f10-b549-7e7fcb829515-kube-api-access-9gkvq\") pod \"maas-api-e2e-explicit-345a2c-7989bf656d-26nx8\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:20.906811 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.906772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb89bc9-7da7-4f10-b549-7e7fcb829515-openshift-service-ca\") pod \"maas-api-e2e-explicit-345a2c-7989bf656d-26nx8\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:20.906811 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.906815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb89bc9-7da7-4f10-b549-7e7fcb829515-maas-api-tls\") pod \"maas-api-e2e-explicit-345a2c-7989bf656d-26nx8\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:20.907057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.906844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eb89bc9-7da7-4f10-b549-7e7fcb829515-metrics-certs\") pod \"maas-api-e2e-explicit-345a2c-7989bf656d-26nx8\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:20.907057 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.906868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gkvq\" (UniqueName: \"kubernetes.io/projected/4eb89bc9-7da7-4f10-b549-7e7fcb829515-kube-api-access-9gkvq\") pod \"maas-api-e2e-explicit-345a2c-7989bf656d-26nx8\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:20.907562 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.907540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb89bc9-7da7-4f10-b549-7e7fcb829515-openshift-service-ca\") pod \"maas-api-e2e-explicit-345a2c-7989bf656d-26nx8\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:20.909485 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.909459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eb89bc9-7da7-4f10-b549-7e7fcb829515-metrics-certs\") pod \"maas-api-e2e-explicit-345a2c-7989bf656d-26nx8\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:20.909599 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.909462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb89bc9-7da7-4f10-b549-7e7fcb829515-maas-api-tls\") pod \"maas-api-e2e-explicit-345a2c-7989bf656d-26nx8\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:20.915527 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:20.915505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gkvq\" (UniqueName: \"kubernetes.io/projected/4eb89bc9-7da7-4f10-b549-7e7fcb829515-kube-api-access-9gkvq\") pod \"maas-api-e2e-explicit-345a2c-7989bf656d-26nx8\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:21.060778 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:21.060686    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:21.353311 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:21.353286    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"]
Sep 10 14:34:21.355760 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:34:21.355721    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb89bc9_7da7_4f10_b549_7e7fcb829515.slice/crio-f3341a1be1769c12367d30a0ea04d7eb98a17491ca2bf9a9d5df7055f01c72dd WatchSource:0}: Error finding container f3341a1be1769c12367d30a0ea04d7eb98a17491ca2bf9a9d5df7055f01c72dd: Status 404 returned error can't find the container with id f3341a1be1769c12367d30a0ea04d7eb98a17491ca2bf9a9d5df7055f01c72dd
Sep 10 14:34:21.522556 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:21.522520    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"]
Sep 10 14:34:21.527212 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:21.527195    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"
Sep 10 14:34:21.530682 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:21.530658    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-345a2c-dockercfg-89fdx\""
Sep 10 14:34:21.530682 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:21.530669    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-345a2c\""
Sep 10 14:34:21.547653 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:21.547625    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"]
Sep 10 14:34:21.614159 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:21.614073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d5c95e-02c6-40a0-a6ef-310b401c7c37-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6\" (UID: \"96d5c95e-02c6-40a0-a6ef-310b401c7c37\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"
Sep 10 14:34:21.614312 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:21.614188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9zt\" (UniqueName: \"kubernetes.io/projected/96d5c95e-02c6-40a0-a6ef-310b401c7c37-kube-api-access-gr9zt\") pod \"payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6\" (UID: \"96d5c95e-02c6-40a0-a6ef-310b401c7c37\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"
Sep 10 14:34:21.714991 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:21.714951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d5c95e-02c6-40a0-a6ef-310b401c7c37-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6\" (UID: \"96d5c95e-02c6-40a0-a6ef-310b401c7c37\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"
Sep 10 14:34:21.715180 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:21.715083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9zt\" (UniqueName: \"kubernetes.io/projected/96d5c95e-02c6-40a0-a6ef-310b401c7c37-kube-api-access-gr9zt\") pod \"payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6\" (UID: \"96d5c95e-02c6-40a0-a6ef-310b401c7c37\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"
Sep 10 14:34:21.715634 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:21.715614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d5c95e-02c6-40a0-a6ef-310b401c7c37-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6\" (UID: \"96d5c95e-02c6-40a0-a6ef-310b401c7c37\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"
Sep 10 14:34:21.730122 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:21.730091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9zt\" (UniqueName: \"kubernetes.io/projected/96d5c95e-02c6-40a0-a6ef-310b401c7c37-kube-api-access-gr9zt\") pod \"payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6\" (UID: \"96d5c95e-02c6-40a0-a6ef-310b401c7c37\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"
Sep 10 14:34:21.837868 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:21.837830    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"
Sep 10 14:34:21.924513 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:21.924485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8" event={"ID":"4eb89bc9-7da7-4f10-b549-7e7fcb829515","Type":"ContainerStarted","Data":"f3341a1be1769c12367d30a0ea04d7eb98a17491ca2bf9a9d5df7055f01c72dd"}
Sep 10 14:34:22.112349 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:22.112244    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"]
Sep 10 14:34:22.114597 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:34:22.114569    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d5c95e_02c6_40a0_a6ef_310b401c7c37.slice/crio-4430341f38d8c7360bc07b58a1cf04c8b4748fb22d94ad9cfb56a7d09735a714 WatchSource:0}: Error finding container 4430341f38d8c7360bc07b58a1cf04c8b4748fb22d94ad9cfb56a7d09735a714: Status 404 returned error can't find the container with id 4430341f38d8c7360bc07b58a1cf04c8b4748fb22d94ad9cfb56a7d09735a714
Sep 10 14:34:22.932405 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:22.932367    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6" event={"ID":"96d5c95e-02c6-40a0-a6ef-310b401c7c37","Type":"ContainerStarted","Data":"b322f96cbd341c20a1673e0f974d3973ffe0c684ca10ae98bd97680a589b4602"}
Sep 10 14:34:22.932823 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:22.932458    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6" event={"ID":"96d5c95e-02c6-40a0-a6ef-310b401c7c37","Type":"ContainerStarted","Data":"4430341f38d8c7360bc07b58a1cf04c8b4748fb22d94ad9cfb56a7d09735a714"}
Sep 10 14:34:22.932823 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:22.932484    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"
Sep 10 14:34:22.934240 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:22.934213    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8" event={"ID":"4eb89bc9-7da7-4f10-b549-7e7fcb829515","Type":"ContainerStarted","Data":"62afe48eb5400073051d13dc3b63b32029da7249280beebba2d0967faa144dd0"}
Sep 10 14:34:22.934352 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:22.934293    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:22.949738 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:22.949686    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6" podStartSLOduration=1.949673059 podStartE2EDuration="1.949673059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:34:22.949009709 +0000 UTC m=+881.140537911" watchObservedRunningTime="2026-09-10 14:34:22.949673059 +0000 UTC m=+881.141201283"
Sep 10 14:34:22.969474 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:22.969421    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8" podStartSLOduration=2.419665392 podStartE2EDuration="2.969406585s" totalImagesPullingTime="549.741193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:34:22.966408211 +0000 UTC m=+881.157936422" watchObservedRunningTime="2026-09-10 14:34:22.969406585 +0000 UTC m=+881.160934816"
Sep 10 14:34:28.945464 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:28.945429    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:34:33.942554 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:33.942521    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"
Sep 10 14:34:42.298085 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:42.298051    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:34:42.301792 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:34:42.301766    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:35:13.508245 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.508204    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"]
Sep 10 14:35:13.511319 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.511297    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.513768 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.513748    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-345a2c\"/\"openshift-service-ca.crt\""
Sep 10 14:35:13.514958 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.514936    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-345a2c\"/\"kube-root-ca.crt\""
Sep 10 14:35:13.515139 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.514961    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-345a2c\"/\"explicit-model-345a2c-kserve-self-signed-certs\""
Sep 10 14:35:13.515139 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.514985    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-345a2c\"/\"default-dockercfg-9knt2\""
Sep 10 14:35:13.523029 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.522999    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"]
Sep 10 14:35:13.596807 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.596766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-tmp-dir\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.597006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.596886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g99d\" (UniqueName: \"kubernetes.io/projected/29b8a203-0f40-4481-8572-aed68a661acd-kube-api-access-5g99d\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.597006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.596917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.597006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.596937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-dshm\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.597006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.596958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-model-cache\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.597153 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.597026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-home\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.698407 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.698365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g99d\" (UniqueName: \"kubernetes.io/projected/29b8a203-0f40-4481-8572-aed68a661acd-kube-api-access-5g99d\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.698407 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.698407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.698651 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.698425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-dshm\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.698651 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.698444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-model-cache\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.698651 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.698465    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-home\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.698651 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.698510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-tmp-dir\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.698916 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.698892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-model-cache\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.699003 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.698914    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-home\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.699003 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.698945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-tmp-dir\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.700664 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.700642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-dshm\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.700921 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.700903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.706511 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.706485    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g99d\" (UniqueName: \"kubernetes.io/projected/29b8a203-0f40-4481-8572-aed68a661acd-kube-api-access-5g99d\") pod \"explicit-model-345a2c-kserve-5dddc65ffc-7dpkv\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") " pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:13.820942 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:13.820857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:14.092374 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:14.092348    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"]
Sep 10 14:35:14.093920 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:35:14.093889    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29b8a203_0f40_4481_8572_aed68a661acd.slice/crio-b435707c2f09f94e1da85475d2bfaff459fdaf9334b5a8db15a7d5ea04b4c527 WatchSource:0}: Error finding container b435707c2f09f94e1da85475d2bfaff459fdaf9334b5a8db15a7d5ea04b4c527: Status 404 returned error can't find the container with id b435707c2f09f94e1da85475d2bfaff459fdaf9334b5a8db15a7d5ea04b4c527
Sep 10 14:35:14.237629 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:14.237586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv" event={"ID":"29b8a203-0f40-4481-8572-aed68a661acd","Type":"ContainerStarted","Data":"dce7776050a3edf3211e7060a1b0c2e295d12e727bfa8a63e64f96e769bd0d0c"}
Sep 10 14:35:14.237817 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:14.237643    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv" event={"ID":"29b8a203-0f40-4481-8572-aed68a661acd","Type":"ContainerStarted","Data":"b435707c2f09f94e1da85475d2bfaff459fdaf9334b5a8db15a7d5ea04b4c527"}
Sep 10 14:35:14.256664 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:14.256609    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv" podStartSLOduration=1.256592295 podStartE2EDuration="1.256592295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:35:14.254894293 +0000 UTC m=+932.446422491" watchObservedRunningTime="2026-09-10 14:35:14.256592295 +0000 UTC m=+932.448120493"
Sep 10 14:35:23.821232 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:23.821190    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:23.821638 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:23.821245    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:23.833802 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:23.833774    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:24.309901 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:24.309869    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:24.646153 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:24.646114    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"]
Sep 10 14:35:25.413026 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:35:25.412989    2580 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-self-signed-certs: secret "explicit-model-345a2c-kserve-self-signed-certs" not found
Sep 10 14:35:25.413432 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:35:25.413073    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs podName:29b8a203-0f40-4481-8572-aed68a661acd nodeName:}" failed. No retries permitted until 2026-09-10 14:35:25.91305231 +0000 UTC m=+944.104580487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs") pod "explicit-model-345a2c-kserve-5dddc65ffc-7dpkv" (UID: "29b8a203-0f40-4481-8572-aed68a661acd") : secret "explicit-model-345a2c-kserve-self-signed-certs" not found
Sep 10 14:35:25.918831 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:35:25.918797    2580 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-self-signed-certs: secret "explicit-model-345a2c-kserve-self-signed-certs" not found
Sep 10 14:35:25.919045 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:35:25.918872    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs podName:29b8a203-0f40-4481-8572-aed68a661acd nodeName:}" failed. No retries permitted until 2026-09-10 14:35:26.91885782 +0000 UTC m=+945.110385997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs") pod "explicit-model-345a2c-kserve-5dddc65ffc-7dpkv" (UID: "29b8a203-0f40-4481-8572-aed68a661acd") : secret "explicit-model-345a2c-kserve-self-signed-certs" not found
Sep 10 14:35:26.930238 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:35:26.930197    2580 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-self-signed-certs: secret "explicit-model-345a2c-kserve-self-signed-certs" not found
Sep 10 14:35:26.930614 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:35:26.930292    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs podName:29b8a203-0f40-4481-8572-aed68a661acd nodeName:}" failed. No retries permitted until 2026-09-10 14:35:28.930275323 +0000 UTC m=+947.121803499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs") pod "explicit-model-345a2c-kserve-5dddc65ffc-7dpkv" (UID: "29b8a203-0f40-4481-8572-aed68a661acd") : secret "explicit-model-345a2c-kserve-self-signed-certs" not found
Sep 10 14:35:28.948933 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:35:28.948901    2580 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-self-signed-certs: secret "explicit-model-345a2c-kserve-self-signed-certs" not found
Sep 10 14:35:28.949335 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:35:28.949002    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs podName:29b8a203-0f40-4481-8572-aed68a661acd nodeName:}" failed. No retries permitted until 2026-09-10 14:35:32.948960524 +0000 UTC m=+951.140488700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs") pod "explicit-model-345a2c-kserve-5dddc65ffc-7dpkv" (UID: "29b8a203-0f40-4481-8572-aed68a661acd") : secret "explicit-model-345a2c-kserve-self-signed-certs" not found
Sep 10 14:35:32.989062 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:35:32.989019    2580 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-self-signed-certs: secret "explicit-model-345a2c-kserve-self-signed-certs" not found
Sep 10 14:35:32.989457 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:35:32.989095    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs podName:29b8a203-0f40-4481-8572-aed68a661acd nodeName:}" failed. No retries permitted until 2026-09-10 14:35:40.989079548 +0000 UTC m=+959.180607724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs") pod "explicit-model-345a2c-kserve-5dddc65ffc-7dpkv" (UID: "29b8a203-0f40-4481-8572-aed68a661acd") : secret "explicit-model-345a2c-kserve-self-signed-certs" not found
Sep 10 14:35:35.294331 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.294298    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"]
Sep 10 14:35:35.294841 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.294560    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8" podUID="4eb89bc9-7da7-4f10-b549-7e7fcb829515" containerName="maas-api" containerID="cri-o://62afe48eb5400073051d13dc3b63b32029da7249280beebba2d0967faa144dd0" gracePeriod=30
Sep 10 14:35:35.353265 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.353233    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"]
Sep 10 14:35:35.353506 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.353459    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6" podUID="96d5c95e-02c6-40a0-a6ef-310b401c7c37" containerName="payload-pre-processing" containerID="cri-o://b322f96cbd341c20a1673e0f974d3973ffe0c684ca10ae98bd97680a589b4602" gracePeriod=30
Sep 10 14:35:35.593814 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.593790    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:35:35.619651 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.619630    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"
Sep 10 14:35:35.717428 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.717383    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb89bc9-7da7-4f10-b549-7e7fcb829515-maas-api-tls\") pod \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") "
Sep 10 14:35:35.717428 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.717435    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gkvq\" (UniqueName: \"kubernetes.io/projected/4eb89bc9-7da7-4f10-b549-7e7fcb829515-kube-api-access-9gkvq\") pod \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") "
Sep 10 14:35:35.717657 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.717467    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr9zt\" (UniqueName: \"kubernetes.io/projected/96d5c95e-02c6-40a0-a6ef-310b401c7c37-kube-api-access-gr9zt\") pod \"96d5c95e-02c6-40a0-a6ef-310b401c7c37\" (UID: \"96d5c95e-02c6-40a0-a6ef-310b401c7c37\") "
Sep 10 14:35:35.717657 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.717565    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb89bc9-7da7-4f10-b549-7e7fcb829515-openshift-service-ca\") pod \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") "
Sep 10 14:35:35.717778 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.717660    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eb89bc9-7da7-4f10-b549-7e7fcb829515-metrics-certs\") pod \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\" (UID: \"4eb89bc9-7da7-4f10-b549-7e7fcb829515\") "
Sep 10 14:35:35.717778 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.717686    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d5c95e-02c6-40a0-a6ef-310b401c7c37-plugins-config-volume\") pod \"96d5c95e-02c6-40a0-a6ef-310b401c7c37\" (UID: \"96d5c95e-02c6-40a0-a6ef-310b401c7c37\") "
Sep 10 14:35:35.718018 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.717934    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb89bc9-7da7-4f10-b549-7e7fcb829515-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4eb89bc9-7da7-4f10-b549-7e7fcb829515" (UID: "4eb89bc9-7da7-4f10-b549-7e7fcb829515"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:35:35.718149 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.718089    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb89bc9-7da7-4f10-b549-7e7fcb829515-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:35:35.718209 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.718168    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d5c95e-02c6-40a0-a6ef-310b401c7c37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96d5c95e-02c6-40a0-a6ef-310b401c7c37" (UID: "96d5c95e-02c6-40a0-a6ef-310b401c7c37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:35:35.719697 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.719677    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb89bc9-7da7-4f10-b549-7e7fcb829515-kube-api-access-9gkvq" (OuterVolumeSpecName: "kube-api-access-9gkvq") pod "4eb89bc9-7da7-4f10-b549-7e7fcb829515" (UID: "4eb89bc9-7da7-4f10-b549-7e7fcb829515"). InnerVolumeSpecName "kube-api-access-9gkvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:35:35.719791 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.719703    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d5c95e-02c6-40a0-a6ef-310b401c7c37-kube-api-access-gr9zt" (OuterVolumeSpecName: "kube-api-access-gr9zt") pod "96d5c95e-02c6-40a0-a6ef-310b401c7c37" (UID: "96d5c95e-02c6-40a0-a6ef-310b401c7c37"). InnerVolumeSpecName "kube-api-access-gr9zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:35:35.719983 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.719940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb89bc9-7da7-4f10-b549-7e7fcb829515-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4eb89bc9-7da7-4f10-b549-7e7fcb829515" (UID: "4eb89bc9-7da7-4f10-b549-7e7fcb829515"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:35:35.720056 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.719965    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb89bc9-7da7-4f10-b549-7e7fcb829515-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4eb89bc9-7da7-4f10-b549-7e7fcb829515" (UID: "4eb89bc9-7da7-4f10-b549-7e7fcb829515"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:35:35.818896 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.818809    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eb89bc9-7da7-4f10-b549-7e7fcb829515-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:35:35.818896 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.818835    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d5c95e-02c6-40a0-a6ef-310b401c7c37-plugins-config-volume\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:35:35.818896 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.818844    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb89bc9-7da7-4f10-b549-7e7fcb829515-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:35:35.818896 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.818854    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gkvq\" (UniqueName: \"kubernetes.io/projected/4eb89bc9-7da7-4f10-b549-7e7fcb829515-kube-api-access-9gkvq\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:35:35.818896 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:35.818863    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9zt\" (UniqueName: \"kubernetes.io/projected/96d5c95e-02c6-40a0-a6ef-310b401c7c37-kube-api-access-gr9zt\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:35:36.369791 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.369751    2580 generic.go:356] "Generic (PLEG): container finished" podID="96d5c95e-02c6-40a0-a6ef-310b401c7c37" containerID="b322f96cbd341c20a1673e0f974d3973ffe0c684ca10ae98bd97680a589b4602" exitCode=0
Sep 10 14:35:36.369791 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.369794    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"
Sep 10 14:35:36.370284 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.369832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6" event={"ID":"96d5c95e-02c6-40a0-a6ef-310b401c7c37","Type":"ContainerDied","Data":"b322f96cbd341c20a1673e0f974d3973ffe0c684ca10ae98bd97680a589b4602"}
Sep 10 14:35:36.370284 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.369891    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6" event={"ID":"96d5c95e-02c6-40a0-a6ef-310b401c7c37","Type":"ContainerDied","Data":"4430341f38d8c7360bc07b58a1cf04c8b4748fb22d94ad9cfb56a7d09735a714"}
Sep 10 14:35:36.370284 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.369912    2580 scope.go:117] "RemoveContainer" containerID="b322f96cbd341c20a1673e0f974d3973ffe0c684ca10ae98bd97680a589b4602"
Sep 10 14:35:36.371371 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.371217    2580 generic.go:356] "Generic (PLEG): container finished" podID="4eb89bc9-7da7-4f10-b549-7e7fcb829515" containerID="62afe48eb5400073051d13dc3b63b32029da7249280beebba2d0967faa144dd0" exitCode=0
Sep 10 14:35:36.371371 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.371287    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"
Sep 10 14:35:36.371371 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.371308    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8" event={"ID":"4eb89bc9-7da7-4f10-b549-7e7fcb829515","Type":"ContainerDied","Data":"62afe48eb5400073051d13dc3b63b32029da7249280beebba2d0967faa144dd0"}
Sep 10 14:35:36.371371 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.371343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8" event={"ID":"4eb89bc9-7da7-4f10-b549-7e7fcb829515","Type":"ContainerDied","Data":"f3341a1be1769c12367d30a0ea04d7eb98a17491ca2bf9a9d5df7055f01c72dd"}
Sep 10 14:35:36.379160 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.379140    2580 scope.go:117] "RemoveContainer" containerID="b322f96cbd341c20a1673e0f974d3973ffe0c684ca10ae98bd97680a589b4602"
Sep 10 14:35:36.379460 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:35:36.379437    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b322f96cbd341c20a1673e0f974d3973ffe0c684ca10ae98bd97680a589b4602\": container with ID starting with b322f96cbd341c20a1673e0f974d3973ffe0c684ca10ae98bd97680a589b4602 not found: ID does not exist" containerID="b322f96cbd341c20a1673e0f974d3973ffe0c684ca10ae98bd97680a589b4602"
Sep 10 14:35:36.379510 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.379471    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b322f96cbd341c20a1673e0f974d3973ffe0c684ca10ae98bd97680a589b4602"} err="failed to get container status \"b322f96cbd341c20a1673e0f974d3973ffe0c684ca10ae98bd97680a589b4602\": rpc error: code = NotFound desc = could not find container \"b322f96cbd341c20a1673e0f974d3973ffe0c684ca10ae98bd97680a589b4602\": container with ID starting with b322f96cbd341c20a1673e0f974d3973ffe0c684ca10ae98bd97680a589b4602 not found: ID does not exist"
Sep 10 14:35:36.379510 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.379489    2580 scope.go:117] "RemoveContainer" containerID="62afe48eb5400073051d13dc3b63b32029da7249280beebba2d0967faa144dd0"
Sep 10 14:35:36.387670 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.387654    2580 scope.go:117] "RemoveContainer" containerID="62afe48eb5400073051d13dc3b63b32029da7249280beebba2d0967faa144dd0"
Sep 10 14:35:36.387995 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:35:36.387950    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62afe48eb5400073051d13dc3b63b32029da7249280beebba2d0967faa144dd0\": container with ID starting with 62afe48eb5400073051d13dc3b63b32029da7249280beebba2d0967faa144dd0 not found: ID does not exist" containerID="62afe48eb5400073051d13dc3b63b32029da7249280beebba2d0967faa144dd0"
Sep 10 14:35:36.388085 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.387997    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62afe48eb5400073051d13dc3b63b32029da7249280beebba2d0967faa144dd0"} err="failed to get container status \"62afe48eb5400073051d13dc3b63b32029da7249280beebba2d0967faa144dd0\": rpc error: code = NotFound desc = could not find container \"62afe48eb5400073051d13dc3b63b32029da7249280beebba2d0967faa144dd0\": container with ID starting with 62afe48eb5400073051d13dc3b63b32029da7249280beebba2d0967faa144dd0 not found: ID does not exist"
Sep 10 14:35:36.391440 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.391417    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"]
Sep 10 14:35:36.400653 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.395916    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-345a2c-7cfd79d666-bndv6"]
Sep 10 14:35:36.413373 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.413344    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"]
Sep 10 14:35:36.416967 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:36.416944    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-345a2c-7989bf656d-26nx8"]
Sep 10 14:35:38.309932 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:38.309900    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eb89bc9-7da7-4f10-b549-7e7fcb829515" path="/var/lib/kubelet/pods/4eb89bc9-7da7-4f10-b549-7e7fcb829515/volumes"
Sep 10 14:35:38.310347 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:38.310331    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96d5c95e-02c6-40a0-a6ef-310b401c7c37" path="/var/lib/kubelet/pods/96d5c95e-02c6-40a0-a6ef-310b401c7c37/volumes"
Sep 10 14:35:41.069635 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:35:41.069599    2580 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-self-signed-certs: secret "explicit-model-345a2c-kserve-self-signed-certs" not found
Sep 10 14:35:41.070071 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:35:41.069674    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs podName:29b8a203-0f40-4481-8572-aed68a661acd nodeName:}" failed. No retries permitted until 2026-09-10 14:35:57.06966123 +0000 UTC m=+975.261189406 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs") pod "explicit-model-345a2c-kserve-5dddc65ffc-7dpkv" (UID: "29b8a203-0f40-4481-8572-aed68a661acd") : secret "explicit-model-345a2c-kserve-self-signed-certs" not found
Sep 10 14:35:41.328220 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.328124    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv" podUID="29b8a203-0f40-4481-8572-aed68a661acd" containerName="main" containerID="cri-o://dce7776050a3edf3211e7060a1b0c2e295d12e727bfa8a63e64f96e769bd0d0c" gracePeriod=45
Sep 10 14:35:41.578343 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.578276    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:41.675223 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.675178    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g99d\" (UniqueName: \"kubernetes.io/projected/29b8a203-0f40-4481-8572-aed68a661acd-kube-api-access-5g99d\") pod \"29b8a203-0f40-4481-8572-aed68a661acd\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") "
Sep 10 14:35:41.675402 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.675239    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-tmp-dir\") pod \"29b8a203-0f40-4481-8572-aed68a661acd\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") "
Sep 10 14:35:41.675402 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.675355    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-dshm\") pod \"29b8a203-0f40-4481-8572-aed68a661acd\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") "
Sep 10 14:35:41.675402 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.675401    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs\") pod \"29b8a203-0f40-4481-8572-aed68a661acd\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") "
Sep 10 14:35:41.675566 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.675431    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-home\") pod \"29b8a203-0f40-4481-8572-aed68a661acd\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") "
Sep 10 14:35:41.675566 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.675457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-model-cache\") pod \"29b8a203-0f40-4481-8572-aed68a661acd\" (UID: \"29b8a203-0f40-4481-8572-aed68a661acd\") "
Sep 10 14:35:41.675566 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.675508    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29b8a203-0f40-4481-8572-aed68a661acd" (UID: "29b8a203-0f40-4481-8572-aed68a661acd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:35:41.675738 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.675714    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-home" (OuterVolumeSpecName: "home") pod "29b8a203-0f40-4481-8572-aed68a661acd" (UID: "29b8a203-0f40-4481-8572-aed68a661acd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:35:41.675808 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.675731    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-model-cache" (OuterVolumeSpecName: "model-cache") pod "29b8a203-0f40-4481-8572-aed68a661acd" (UID: "29b8a203-0f40-4481-8572-aed68a661acd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:35:41.675871 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.675851    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:35:41.675914 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.675878    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:35:41.675914 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.675892    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:35:41.677601 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.677579    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-dshm" (OuterVolumeSpecName: "dshm") pod "29b8a203-0f40-4481-8572-aed68a661acd" (UID: "29b8a203-0f40-4481-8572-aed68a661acd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:35:41.677601 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.677586    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29b8a203-0f40-4481-8572-aed68a661acd" (UID: "29b8a203-0f40-4481-8572-aed68a661acd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:35:41.677749 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.677629    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b8a203-0f40-4481-8572-aed68a661acd-kube-api-access-5g99d" (OuterVolumeSpecName: "kube-api-access-5g99d") pod "29b8a203-0f40-4481-8572-aed68a661acd" (UID: "29b8a203-0f40-4481-8572-aed68a661acd"). InnerVolumeSpecName "kube-api-access-5g99d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:35:41.776436 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.776385    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b8a203-0f40-4481-8572-aed68a661acd-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:35:41.776436 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.776427    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b8a203-0f40-4481-8572-aed68a661acd-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:35:41.776436 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:41.776439    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g99d\" (UniqueName: \"kubernetes.io/projected/29b8a203-0f40-4481-8572-aed68a661acd-kube-api-access-5g99d\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:35:42.408404 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:42.408368    2580 generic.go:356] "Generic (PLEG): container finished" podID="29b8a203-0f40-4481-8572-aed68a661acd" containerID="dce7776050a3edf3211e7060a1b0c2e295d12e727bfa8a63e64f96e769bd0d0c" exitCode=0
Sep 10 14:35:42.408769 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:42.408431    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"
Sep 10 14:35:42.408769 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:42.408457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv" event={"ID":"29b8a203-0f40-4481-8572-aed68a661acd","Type":"ContainerDied","Data":"dce7776050a3edf3211e7060a1b0c2e295d12e727bfa8a63e64f96e769bd0d0c"}
Sep 10 14:35:42.408769 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:42.408505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv" event={"ID":"29b8a203-0f40-4481-8572-aed68a661acd","Type":"ContainerDied","Data":"b435707c2f09f94e1da85475d2bfaff459fdaf9334b5a8db15a7d5ea04b4c527"}
Sep 10 14:35:42.408769 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:42.408532    2580 scope.go:117] "RemoveContainer" containerID="dce7776050a3edf3211e7060a1b0c2e295d12e727bfa8a63e64f96e769bd0d0c"
Sep 10 14:35:42.417282 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:42.417261    2580 scope.go:117] "RemoveContainer" containerID="dce7776050a3edf3211e7060a1b0c2e295d12e727bfa8a63e64f96e769bd0d0c"
Sep 10 14:35:42.417569 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:35:42.417548    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce7776050a3edf3211e7060a1b0c2e295d12e727bfa8a63e64f96e769bd0d0c\": container with ID starting with dce7776050a3edf3211e7060a1b0c2e295d12e727bfa8a63e64f96e769bd0d0c not found: ID does not exist" containerID="dce7776050a3edf3211e7060a1b0c2e295d12e727bfa8a63e64f96e769bd0d0c"
Sep 10 14:35:42.417616 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:42.417578    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce7776050a3edf3211e7060a1b0c2e295d12e727bfa8a63e64f96e769bd0d0c"} err="failed to get container status \"dce7776050a3edf3211e7060a1b0c2e295d12e727bfa8a63e64f96e769bd0d0c\": rpc error: code = NotFound desc = could not find container \"dce7776050a3edf3211e7060a1b0c2e295d12e727bfa8a63e64f96e769bd0d0c\": container with ID starting with dce7776050a3edf3211e7060a1b0c2e295d12e727bfa8a63e64f96e769bd0d0c not found: ID does not exist"
Sep 10 14:35:42.424683 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:42.424660    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"]
Sep 10 14:35:42.428108 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:42.428086    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-345a2c/explicit-model-345a2c-kserve-5dddc65ffc-7dpkv"]
Sep 10 14:35:44.310026 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:44.309966    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29b8a203-0f40-4481-8572-aed68a661acd" path="/var/lib/kubelet/pods/29b8a203-0f40-4481-8572-aed68a661acd/volumes"
Sep 10 14:35:47.330135 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.330099    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"]
Sep 10 14:35:47.330719 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.330699    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d5c95e-02c6-40a0-a6ef-310b401c7c37" containerName="payload-pre-processing"
Sep 10 14:35:47.330804 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.330722    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d5c95e-02c6-40a0-a6ef-310b401c7c37" containerName="payload-pre-processing"
Sep 10 14:35:47.330804 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.330747    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb89bc9-7da7-4f10-b549-7e7fcb829515" containerName="maas-api"
Sep 10 14:35:47.330804 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.330756    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb89bc9-7da7-4f10-b549-7e7fcb829515" containerName="maas-api"
Sep 10 14:35:47.330804 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.330771    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29b8a203-0f40-4481-8572-aed68a661acd" containerName="main"
Sep 10 14:35:47.330804 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.330780    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29b8a203-0f40-4481-8572-aed68a661acd" containerName="main"
Sep 10 14:35:47.331069 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.330880    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29b8a203-0f40-4481-8572-aed68a661acd" containerName="main"
Sep 10 14:35:47.331069 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.330894    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d5c95e-02c6-40a0-a6ef-310b401c7c37" containerName="payload-pre-processing"
Sep 10 14:35:47.331069 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.330907    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eb89bc9-7da7-4f10-b549-7e7fcb829515" containerName="maas-api"
Sep 10 14:35:47.335695 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.335672    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:35:47.338844 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.338816    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e25a5e89-serving-cert\""
Sep 10 14:35:47.361900 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.361861    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"]
Sep 10 14:35:47.426209 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.426176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca45038-4f10-4faa-bd93-b80313a25964-metrics-certs\") pod \"maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:35:47.426417 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.426226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdwr5\" (UniqueName: \"kubernetes.io/projected/7ca45038-4f10-4faa-bd93-b80313a25964-kube-api-access-kdwr5\") pod \"maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:35:47.426417 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.426261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca45038-4f10-4faa-bd93-b80313a25964-openshift-service-ca\") pod \"maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:35:47.426417 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.426289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca45038-4f10-4faa-bd93-b80313a25964-maas-api-tls\") pod \"maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:35:47.527038 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.527001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca45038-4f10-4faa-bd93-b80313a25964-metrics-certs\") pod \"maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:35:47.527215 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.527054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdwr5\" (UniqueName: \"kubernetes.io/projected/7ca45038-4f10-4faa-bd93-b80313a25964-kube-api-access-kdwr5\") pod \"maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:35:47.527215 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.527078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca45038-4f10-4faa-bd93-b80313a25964-openshift-service-ca\") pod \"maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:35:47.527215 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.527101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca45038-4f10-4faa-bd93-b80313a25964-maas-api-tls\") pod \"maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:35:47.527876 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.527849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca45038-4f10-4faa-bd93-b80313a25964-openshift-service-ca\") pod \"maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:35:47.529600 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.529577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca45038-4f10-4faa-bd93-b80313a25964-metrics-certs\") pod \"maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:35:47.529698 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.529613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca45038-4f10-4faa-bd93-b80313a25964-maas-api-tls\") pod \"maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:35:47.536933 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.536913    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdwr5\" (UniqueName: \"kubernetes.io/projected/7ca45038-4f10-4faa-bd93-b80313a25964-kube-api-access-kdwr5\") pod \"maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:35:47.646832 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.646804    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:35:47.925913 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.925879    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"]
Sep 10 14:35:47.928283 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:35:47.928251    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca45038_4f10_4faa_bd93_b80313a25964.slice/crio-87625646efdbb817f2b2e557032c4ebbbb599a7a9a715f8329d1729e1926d2d8 WatchSource:0}: Error finding container 87625646efdbb817f2b2e557032c4ebbbb599a7a9a715f8329d1729e1926d2d8: Status 404 returned error can't find the container with id 87625646efdbb817f2b2e557032c4ebbbb599a7a9a715f8329d1729e1926d2d8
Sep 10 14:35:47.929751 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:47.929726    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:35:48.085676 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:48.085641    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"]
Sep 10 14:35:48.090357 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:48.090339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"
Sep 10 14:35:48.094499 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:48.094474    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e25a5e89-dockercfg-jqxsl\""
Sep 10 14:35:48.094660 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:48.094640    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e25a5e89\""
Sep 10 14:35:48.107738 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:48.107708    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"]
Sep 10 14:35:48.131672 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:48.131638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8a57a0f-6eb2-4f16-bcab-cc01c96a5640-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp\" (UID: \"d8a57a0f-6eb2-4f16-bcab-cc01c96a5640\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"
Sep 10 14:35:48.131863 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:48.131720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v59d\" (UniqueName: \"kubernetes.io/projected/d8a57a0f-6eb2-4f16-bcab-cc01c96a5640-kube-api-access-4v59d\") pod \"payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp\" (UID: \"d8a57a0f-6eb2-4f16-bcab-cc01c96a5640\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"
Sep 10 14:35:48.232645 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:48.232555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8a57a0f-6eb2-4f16-bcab-cc01c96a5640-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp\" (UID: \"d8a57a0f-6eb2-4f16-bcab-cc01c96a5640\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"
Sep 10 14:35:48.232820 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:48.232648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v59d\" (UniqueName: \"kubernetes.io/projected/d8a57a0f-6eb2-4f16-bcab-cc01c96a5640-kube-api-access-4v59d\") pod \"payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp\" (UID: \"d8a57a0f-6eb2-4f16-bcab-cc01c96a5640\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"
Sep 10 14:35:48.233324 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:48.233303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8a57a0f-6eb2-4f16-bcab-cc01c96a5640-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp\" (UID: \"d8a57a0f-6eb2-4f16-bcab-cc01c96a5640\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"
Sep 10 14:35:48.240647 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:48.240614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v59d\" (UniqueName: \"kubernetes.io/projected/d8a57a0f-6eb2-4f16-bcab-cc01c96a5640-kube-api-access-4v59d\") pod \"payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp\" (UID: \"d8a57a0f-6eb2-4f16-bcab-cc01c96a5640\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"
Sep 10 14:35:48.400198 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:48.400141    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"
Sep 10 14:35:48.446319 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:48.446288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx" event={"ID":"7ca45038-4f10-4faa-bd93-b80313a25964","Type":"ContainerStarted","Data":"87625646efdbb817f2b2e557032c4ebbbb599a7a9a715f8329d1729e1926d2d8"}
Sep 10 14:35:48.682560 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:48.682526    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"]
Sep 10 14:35:48.683956 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:35:48.683913    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a57a0f_6eb2_4f16_bcab_cc01c96a5640.slice/crio-310b545e0d36321810258626c8218fcf21aebb708bd725e8ba3484a097d8773b WatchSource:0}: Error finding container 310b545e0d36321810258626c8218fcf21aebb708bd725e8ba3484a097d8773b: Status 404 returned error can't find the container with id 310b545e0d36321810258626c8218fcf21aebb708bd725e8ba3484a097d8773b
Sep 10 14:35:49.453834 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:49.453799    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx" event={"ID":"7ca45038-4f10-4faa-bd93-b80313a25964","Type":"ContainerStarted","Data":"f3d1192b2e4982bf96d6c8187381c588eae595b9c627ef362ee9a0e6ea856a99"}
Sep 10 14:35:49.454273 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:49.453869    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:35:49.455411 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:49.455384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp" event={"ID":"d8a57a0f-6eb2-4f16-bcab-cc01c96a5640","Type":"ContainerStarted","Data":"8482af24845e4da68143fb374ba14232b300cca2243279a49ee78a354eeeebea"}
Sep 10 14:35:49.455538 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:49.455434    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp" event={"ID":"d8a57a0f-6eb2-4f16-bcab-cc01c96a5640","Type":"ContainerStarted","Data":"310b545e0d36321810258626c8218fcf21aebb708bd725e8ba3484a097d8773b"}
Sep 10 14:35:49.455538 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:49.455462    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"
Sep 10 14:35:49.471341 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:49.471296    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx" podStartSLOduration=1.969814901 podStartE2EDuration="2.471282002s" totalImagesPullingTime="501.467101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:35:49.469349739 +0000 UTC m=+967.660877938" watchObservedRunningTime="2026-09-10 14:35:49.471282002 +0000 UTC m=+967.662810201"
Sep 10 14:35:49.487725 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:49.487675    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp" podStartSLOduration=1.487661183 podStartE2EDuration="1.487661183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:35:49.484235409 +0000 UTC m=+967.675763611" watchObservedRunningTime="2026-09-10 14:35:49.487661183 +0000 UTC m=+967.679189384"
Sep 10 14:35:55.476346 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:35:55.476317    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:36:00.473381 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:00.473346    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"
Sep 10 14:36:09.227515 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.227439    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"]
Sep 10 14:36:09.231494 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.231470    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.234352 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.234323    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-651c34\"/\"kube-root-ca.crt\""
Sep 10 14:36:09.234511 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.234490    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-651c34\"/\"openshift-service-ca.crt\""
Sep 10 14:36:09.235478 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.235456    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-651c34\"/\"default-dockercfg-5r57p\""
Sep 10 14:36:09.235569 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.235504    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-651c34\"/\"noten-model-651c34-kserve-self-signed-certs\""
Sep 10 14:36:09.240525 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.240499    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"]
Sep 10 14:36:09.315858 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.315829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-home\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.315858 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.315860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xqzn\" (UniqueName: \"kubernetes.io/projected/d124e537-f915-4b7f-ad3e-79186cdd8380-kube-api-access-5xqzn\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.316123 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.315880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-tmp-dir\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.316123 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.315913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d124e537-f915-4b7f-ad3e-79186cdd8380-tls-certs\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.316123 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.315967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-dshm\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.316123 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.316037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-model-cache\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.416464 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.416431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d124e537-f915-4b7f-ad3e-79186cdd8380-tls-certs\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.416644 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.416481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-dshm\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.416644 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.416592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-model-cache\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.416776 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.416716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-home\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.416776 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.416750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xqzn\" (UniqueName: \"kubernetes.io/projected/d124e537-f915-4b7f-ad3e-79186cdd8380-kube-api-access-5xqzn\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.416894 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.416776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-tmp-dir\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.417210 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.417178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-tmp-dir\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.417336 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.417221    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-model-cache\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.417336 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.417326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-home\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.419368 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.419340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-dshm\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.419605 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.419588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d124e537-f915-4b7f-ad3e-79186cdd8380-tls-certs\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.427500 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.427480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xqzn\" (UniqueName: \"kubernetes.io/projected/d124e537-f915-4b7f-ad3e-79186cdd8380-kube-api-access-5xqzn\") pod \"noten-model-651c34-kserve-f4fc86d59-sx7cm\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") " pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.543035 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.542921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:09.822099 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:09.822076    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"]
Sep 10 14:36:09.824173 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:36:09.824145    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd124e537_f915_4b7f_ad3e_79186cdd8380.slice/crio-162b68f4aca76e74e04beea190eb12ff2986c34b539cfb0919ec07e4af3d724f WatchSource:0}: Error finding container 162b68f4aca76e74e04beea190eb12ff2986c34b539cfb0919ec07e4af3d724f: Status 404 returned error can't find the container with id 162b68f4aca76e74e04beea190eb12ff2986c34b539cfb0919ec07e4af3d724f
Sep 10 14:36:10.583196 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:10.583154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm" event={"ID":"d124e537-f915-4b7f-ad3e-79186cdd8380","Type":"ContainerStarted","Data":"4bfe977bc6d198881a164924709944898004cfa0158c4f76f26170ca92aa1520"}
Sep 10 14:36:10.583575 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:10.583208    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm" event={"ID":"d124e537-f915-4b7f-ad3e-79186cdd8380","Type":"ContainerStarted","Data":"162b68f4aca76e74e04beea190eb12ff2986c34b539cfb0919ec07e4af3d724f"}
Sep 10 14:36:10.603569 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:10.603521    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm" podStartSLOduration=1.603503974 podStartE2EDuration="1.603503974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:36:10.601523506 +0000 UTC m=+988.793051704" watchObservedRunningTime="2026-09-10 14:36:10.603503974 +0000 UTC m=+988.795032207"
Sep 10 14:36:19.543709 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:19.543664    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:19.544106 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:19.543730    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:19.556162 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:19.556136    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:19.644343 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:19.644318    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:23.393651 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:23.393601    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"]
Sep 10 14:36:29.221458 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.221418    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"]
Sep 10 14:36:29.221914 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.221717    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx" podUID="7ca45038-4f10-4faa-bd93-b80313a25964" containerName="maas-api" containerID="cri-o://f3d1192b2e4982bf96d6c8187381c588eae595b9c627ef362ee9a0e6ea856a99" gracePeriod=30
Sep 10 14:36:29.289182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.289148    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"]
Sep 10 14:36:29.289374 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.289354    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp" podUID="d8a57a0f-6eb2-4f16-bcab-cc01c96a5640" containerName="payload-pre-processing" containerID="cri-o://8482af24845e4da68143fb374ba14232b300cca2243279a49ee78a354eeeebea" gracePeriod=30
Sep 10 14:36:29.562328 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.562300    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:36:29.564246 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.564228    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"
Sep 10 14:36:29.620427 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.620394    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca45038-4f10-4faa-bd93-b80313a25964-maas-api-tls\") pod \"7ca45038-4f10-4faa-bd93-b80313a25964\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") "
Sep 10 14:36:29.620569 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.620443    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca45038-4f10-4faa-bd93-b80313a25964-metrics-certs\") pod \"7ca45038-4f10-4faa-bd93-b80313a25964\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") "
Sep 10 14:36:29.620622 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.620562    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8a57a0f-6eb2-4f16-bcab-cc01c96a5640-plugins-config-volume\") pod \"d8a57a0f-6eb2-4f16-bcab-cc01c96a5640\" (UID: \"d8a57a0f-6eb2-4f16-bcab-cc01c96a5640\") "
Sep 10 14:36:29.620622 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.620598    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca45038-4f10-4faa-bd93-b80313a25964-openshift-service-ca\") pod \"7ca45038-4f10-4faa-bd93-b80313a25964\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") "
Sep 10 14:36:29.620702 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.620650    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4v59d\" (UniqueName: \"kubernetes.io/projected/d8a57a0f-6eb2-4f16-bcab-cc01c96a5640-kube-api-access-4v59d\") pod \"d8a57a0f-6eb2-4f16-bcab-cc01c96a5640\" (UID: \"d8a57a0f-6eb2-4f16-bcab-cc01c96a5640\") "
Sep 10 14:36:29.620752 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.620720    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdwr5\" (UniqueName: \"kubernetes.io/projected/7ca45038-4f10-4faa-bd93-b80313a25964-kube-api-access-kdwr5\") pod \"7ca45038-4f10-4faa-bd93-b80313a25964\" (UID: \"7ca45038-4f10-4faa-bd93-b80313a25964\") "
Sep 10 14:36:29.621028 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.620986    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca45038-4f10-4faa-bd93-b80313a25964-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ca45038-4f10-4faa-bd93-b80313a25964" (UID: "7ca45038-4f10-4faa-bd93-b80313a25964"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:36:29.621028 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.621000    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a57a0f-6eb2-4f16-bcab-cc01c96a5640-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8a57a0f-6eb2-4f16-bcab-cc01c96a5640" (UID: "d8a57a0f-6eb2-4f16-bcab-cc01c96a5640"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:36:29.621262 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.621115    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8a57a0f-6eb2-4f16-bcab-cc01c96a5640-plugins-config-volume\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:29.621262 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.621135    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca45038-4f10-4faa-bd93-b80313a25964-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:29.622766 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.622745    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca45038-4f10-4faa-bd93-b80313a25964-kube-api-access-kdwr5" (OuterVolumeSpecName: "kube-api-access-kdwr5") pod "7ca45038-4f10-4faa-bd93-b80313a25964" (UID: "7ca45038-4f10-4faa-bd93-b80313a25964"). InnerVolumeSpecName "kube-api-access-kdwr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:36:29.623106 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.623081    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca45038-4f10-4faa-bd93-b80313a25964-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ca45038-4f10-4faa-bd93-b80313a25964" (UID: "7ca45038-4f10-4faa-bd93-b80313a25964"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:36:29.623106 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.623098    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca45038-4f10-4faa-bd93-b80313a25964-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7ca45038-4f10-4faa-bd93-b80313a25964" (UID: "7ca45038-4f10-4faa-bd93-b80313a25964"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:36:29.623218 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.623119    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a57a0f-6eb2-4f16-bcab-cc01c96a5640-kube-api-access-4v59d" (OuterVolumeSpecName: "kube-api-access-4v59d") pod "d8a57a0f-6eb2-4f16-bcab-cc01c96a5640" (UID: "d8a57a0f-6eb2-4f16-bcab-cc01c96a5640"). InnerVolumeSpecName "kube-api-access-4v59d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:36:29.698935 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.698897    2580 generic.go:356] "Generic (PLEG): container finished" podID="d8a57a0f-6eb2-4f16-bcab-cc01c96a5640" containerID="8482af24845e4da68143fb374ba14232b300cca2243279a49ee78a354eeeebea" exitCode=0
Sep 10 14:36:29.699143 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.698949    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"
Sep 10 14:36:29.699143 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.699006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp" event={"ID":"d8a57a0f-6eb2-4f16-bcab-cc01c96a5640","Type":"ContainerDied","Data":"8482af24845e4da68143fb374ba14232b300cca2243279a49ee78a354eeeebea"}
Sep 10 14:36:29.699143 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.699067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp" event={"ID":"d8a57a0f-6eb2-4f16-bcab-cc01c96a5640","Type":"ContainerDied","Data":"310b545e0d36321810258626c8218fcf21aebb708bd725e8ba3484a097d8773b"}
Sep 10 14:36:29.699143 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.699090    2580 scope.go:117] "RemoveContainer" containerID="8482af24845e4da68143fb374ba14232b300cca2243279a49ee78a354eeeebea"
Sep 10 14:36:29.700439 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.700415    2580 generic.go:356] "Generic (PLEG): container finished" podID="7ca45038-4f10-4faa-bd93-b80313a25964" containerID="f3d1192b2e4982bf96d6c8187381c588eae595b9c627ef362ee9a0e6ea856a99" exitCode=0
Sep 10 14:36:29.700580 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.700473    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"
Sep 10 14:36:29.700580 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.700475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx" event={"ID":"7ca45038-4f10-4faa-bd93-b80313a25964","Type":"ContainerDied","Data":"f3d1192b2e4982bf96d6c8187381c588eae595b9c627ef362ee9a0e6ea856a99"}
Sep 10 14:36:29.700580 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.700575    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx" event={"ID":"7ca45038-4f10-4faa-bd93-b80313a25964","Type":"ContainerDied","Data":"87625646efdbb817f2b2e557032c4ebbbb599a7a9a715f8329d1729e1926d2d8"}
Sep 10 14:36:29.711145 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.711123    2580 scope.go:117] "RemoveContainer" containerID="8482af24845e4da68143fb374ba14232b300cca2243279a49ee78a354eeeebea"
Sep 10 14:36:29.711794 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:36:29.711769    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8482af24845e4da68143fb374ba14232b300cca2243279a49ee78a354eeeebea\": container with ID starting with 8482af24845e4da68143fb374ba14232b300cca2243279a49ee78a354eeeebea not found: ID does not exist" containerID="8482af24845e4da68143fb374ba14232b300cca2243279a49ee78a354eeeebea"
Sep 10 14:36:29.711885 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.711801    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8482af24845e4da68143fb374ba14232b300cca2243279a49ee78a354eeeebea"} err="failed to get container status \"8482af24845e4da68143fb374ba14232b300cca2243279a49ee78a354eeeebea\": rpc error: code = NotFound desc = could not find container \"8482af24845e4da68143fb374ba14232b300cca2243279a49ee78a354eeeebea\": container with ID starting with 8482af24845e4da68143fb374ba14232b300cca2243279a49ee78a354eeeebea not found: ID does not exist"
Sep 10 14:36:29.711885 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.711817    2580 scope.go:117] "RemoveContainer" containerID="f3d1192b2e4982bf96d6c8187381c588eae595b9c627ef362ee9a0e6ea856a99"
Sep 10 14:36:29.720032 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.720015    2580 scope.go:117] "RemoveContainer" containerID="f3d1192b2e4982bf96d6c8187381c588eae595b9c627ef362ee9a0e6ea856a99"
Sep 10 14:36:29.720305 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:36:29.720287    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3d1192b2e4982bf96d6c8187381c588eae595b9c627ef362ee9a0e6ea856a99\": container with ID starting with f3d1192b2e4982bf96d6c8187381c588eae595b9c627ef362ee9a0e6ea856a99 not found: ID does not exist" containerID="f3d1192b2e4982bf96d6c8187381c588eae595b9c627ef362ee9a0e6ea856a99"
Sep 10 14:36:29.720352 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.720313    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d1192b2e4982bf96d6c8187381c588eae595b9c627ef362ee9a0e6ea856a99"} err="failed to get container status \"f3d1192b2e4982bf96d6c8187381c588eae595b9c627ef362ee9a0e6ea856a99\": rpc error: code = NotFound desc = could not find container \"f3d1192b2e4982bf96d6c8187381c588eae595b9c627ef362ee9a0e6ea856a99\": container with ID starting with f3d1192b2e4982bf96d6c8187381c588eae595b9c627ef362ee9a0e6ea856a99 not found: ID does not exist"
Sep 10 14:36:29.721736 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.721719    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v59d\" (UniqueName: \"kubernetes.io/projected/d8a57a0f-6eb2-4f16-bcab-cc01c96a5640-kube-api-access-4v59d\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:29.721776 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.721739    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdwr5\" (UniqueName: \"kubernetes.io/projected/7ca45038-4f10-4faa-bd93-b80313a25964-kube-api-access-kdwr5\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:29.721776 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.721748    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca45038-4f10-4faa-bd93-b80313a25964-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:29.721776 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.721756    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca45038-4f10-4faa-bd93-b80313a25964-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:29.725709 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.725687    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"]
Sep 10 14:36:29.729378 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.729356    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e25a5e89-577f5f46cf-rx5lx"]
Sep 10 14:36:29.738465 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.738445    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"]
Sep 10 14:36:29.745448 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:29.745396    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e25a5e89-698f5d6b54-kn2tp"]
Sep 10 14:36:30.316548 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:30.316516    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ca45038-4f10-4faa-bd93-b80313a25964" path="/var/lib/kubelet/pods/7ca45038-4f10-4faa-bd93-b80313a25964/volumes"
Sep 10 14:36:30.316910 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:30.316889    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8a57a0f-6eb2-4f16-bcab-cc01c96a5640" path="/var/lib/kubelet/pods/d8a57a0f-6eb2-4f16-bcab-cc01c96a5640/volumes"
Sep 10 14:36:33.109773 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.109738    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"]
Sep 10 14:36:33.110258 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.110000    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc" podUID="98aad721-07cf-460e-b480-c72053cc063d" containerName="maas-api" containerID="cri-o://04339c14ad255e1dc1334b010e56b0659111a06b832ce9183b3dcfdd1a6984f7" gracePeriod=30
Sep 10 14:36:33.351920 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.351892    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:36:33.458062 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.458019    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll6cm\" (UniqueName: \"kubernetes.io/projected/98aad721-07cf-460e-b480-c72053cc063d-kube-api-access-ll6cm\") pod \"98aad721-07cf-460e-b480-c72053cc063d\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") "
Sep 10 14:36:33.458062 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.458082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98aad721-07cf-460e-b480-c72053cc063d-openshift-service-ca\") pod \"98aad721-07cf-460e-b480-c72053cc063d\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") "
Sep 10 14:36:33.458301 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.458127    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98aad721-07cf-460e-b480-c72053cc063d-metrics-certs\") pod \"98aad721-07cf-460e-b480-c72053cc063d\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") "
Sep 10 14:36:33.458301 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.458144    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98aad721-07cf-460e-b480-c72053cc063d-maas-api-tls\") pod \"98aad721-07cf-460e-b480-c72053cc063d\" (UID: \"98aad721-07cf-460e-b480-c72053cc063d\") "
Sep 10 14:36:33.458521 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.458489    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98aad721-07cf-460e-b480-c72053cc063d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "98aad721-07cf-460e-b480-c72053cc063d" (UID: "98aad721-07cf-460e-b480-c72053cc063d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:36:33.460194 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.460171    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98aad721-07cf-460e-b480-c72053cc063d-kube-api-access-ll6cm" (OuterVolumeSpecName: "kube-api-access-ll6cm") pod "98aad721-07cf-460e-b480-c72053cc063d" (UID: "98aad721-07cf-460e-b480-c72053cc063d"). InnerVolumeSpecName "kube-api-access-ll6cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:36:33.460292 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.460230    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98aad721-07cf-460e-b480-c72053cc063d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "98aad721-07cf-460e-b480-c72053cc063d" (UID: "98aad721-07cf-460e-b480-c72053cc063d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:36:33.460352 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.460321    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98aad721-07cf-460e-b480-c72053cc063d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "98aad721-07cf-460e-b480-c72053cc063d" (UID: "98aad721-07cf-460e-b480-c72053cc063d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:36:33.559198 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.559153    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll6cm\" (UniqueName: \"kubernetes.io/projected/98aad721-07cf-460e-b480-c72053cc063d-kube-api-access-ll6cm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:33.559198 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.559191    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98aad721-07cf-460e-b480-c72053cc063d-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:33.559198 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.559203    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98aad721-07cf-460e-b480-c72053cc063d-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:33.559525 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.559215    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98aad721-07cf-460e-b480-c72053cc063d-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:33.725888 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.725796    2580 generic.go:356] "Generic (PLEG): container finished" podID="98aad721-07cf-460e-b480-c72053cc063d" containerID="04339c14ad255e1dc1334b010e56b0659111a06b832ce9183b3dcfdd1a6984f7" exitCode=0
Sep 10 14:36:33.725888 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.725840    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"
Sep 10 14:36:33.725888 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.725883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc" event={"ID":"98aad721-07cf-460e-b480-c72053cc063d","Type":"ContainerDied","Data":"04339c14ad255e1dc1334b010e56b0659111a06b832ce9183b3dcfdd1a6984f7"}
Sep 10 14:36:33.726144 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.725930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc" event={"ID":"98aad721-07cf-460e-b480-c72053cc063d","Type":"ContainerDied","Data":"019c286e4d9f5604e361492b0a3a46b2f537a647a57f2dc13541c536273fc7d6"}
Sep 10 14:36:33.726144 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.725953    2580 scope.go:117] "RemoveContainer" containerID="04339c14ad255e1dc1334b010e56b0659111a06b832ce9183b3dcfdd1a6984f7"
Sep 10 14:36:33.735711 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.735696    2580 scope.go:117] "RemoveContainer" containerID="04339c14ad255e1dc1334b010e56b0659111a06b832ce9183b3dcfdd1a6984f7"
Sep 10 14:36:33.736017 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:36:33.735998    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04339c14ad255e1dc1334b010e56b0659111a06b832ce9183b3dcfdd1a6984f7\": container with ID starting with 04339c14ad255e1dc1334b010e56b0659111a06b832ce9183b3dcfdd1a6984f7 not found: ID does not exist" containerID="04339c14ad255e1dc1334b010e56b0659111a06b832ce9183b3dcfdd1a6984f7"
Sep 10 14:36:33.736079 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.736026    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04339c14ad255e1dc1334b010e56b0659111a06b832ce9183b3dcfdd1a6984f7"} err="failed to get container status \"04339c14ad255e1dc1334b010e56b0659111a06b832ce9183b3dcfdd1a6984f7\": rpc error: code = NotFound desc = could not find container \"04339c14ad255e1dc1334b010e56b0659111a06b832ce9183b3dcfdd1a6984f7\": container with ID starting with 04339c14ad255e1dc1334b010e56b0659111a06b832ce9183b3dcfdd1a6984f7 not found: ID does not exist"
Sep 10 14:36:33.751805 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.751779    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"]
Sep 10 14:36:33.758162 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:33.758139    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-600197-684694cf78-9l5zc"]
Sep 10 14:36:34.310046 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:34.310012    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98aad721-07cf-460e-b480-c72053cc063d" path="/var/lib/kubelet/pods/98aad721-07cf-460e-b480-c72053cc063d/volumes"
Sep 10 14:36:38.414208 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.414150    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm" podUID="d124e537-f915-4b7f-ad3e-79186cdd8380" containerName="main" containerID="cri-o://4bfe977bc6d198881a164924709944898004cfa0158c4f76f26170ca92aa1520" gracePeriod=45
Sep 10 14:36:38.662237 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.662207    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:38.707563 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.707486    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-model-cache\") pod \"d124e537-f915-4b7f-ad3e-79186cdd8380\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") "
Sep 10 14:36:38.707563 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.707557    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-tmp-dir\") pod \"d124e537-f915-4b7f-ad3e-79186cdd8380\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") "
Sep 10 14:36:38.707786 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.707601    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xqzn\" (UniqueName: \"kubernetes.io/projected/d124e537-f915-4b7f-ad3e-79186cdd8380-kube-api-access-5xqzn\") pod \"d124e537-f915-4b7f-ad3e-79186cdd8380\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") "
Sep 10 14:36:38.707786 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.707624    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-dshm\") pod \"d124e537-f915-4b7f-ad3e-79186cdd8380\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") "
Sep 10 14:36:38.707786 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.707750    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d124e537-f915-4b7f-ad3e-79186cdd8380-tls-certs\") pod \"d124e537-f915-4b7f-ad3e-79186cdd8380\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") "
Sep 10 14:36:38.707942 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.707821    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-home\") pod \"d124e537-f915-4b7f-ad3e-79186cdd8380\" (UID: \"d124e537-f915-4b7f-ad3e-79186cdd8380\") "
Sep 10 14:36:38.707942 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.707811    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-model-cache" (OuterVolumeSpecName: "model-cache") pod "d124e537-f915-4b7f-ad3e-79186cdd8380" (UID: "d124e537-f915-4b7f-ad3e-79186cdd8380"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:36:38.707942 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.707875    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d124e537-f915-4b7f-ad3e-79186cdd8380" (UID: "d124e537-f915-4b7f-ad3e-79186cdd8380"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:36:38.708135 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.708118    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:38.708135 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.708122    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-home" (OuterVolumeSpecName: "home") pod "d124e537-f915-4b7f-ad3e-79186cdd8380" (UID: "d124e537-f915-4b7f-ad3e-79186cdd8380"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:36:38.708235 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.708136    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:38.709859 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.709830    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-dshm" (OuterVolumeSpecName: "dshm") pod "d124e537-f915-4b7f-ad3e-79186cdd8380" (UID: "d124e537-f915-4b7f-ad3e-79186cdd8380"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:36:38.710010 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.709883    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d124e537-f915-4b7f-ad3e-79186cdd8380-kube-api-access-5xqzn" (OuterVolumeSpecName: "kube-api-access-5xqzn") pod "d124e537-f915-4b7f-ad3e-79186cdd8380" (UID: "d124e537-f915-4b7f-ad3e-79186cdd8380"). InnerVolumeSpecName "kube-api-access-5xqzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:36:38.710010 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.709922    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d124e537-f915-4b7f-ad3e-79186cdd8380-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d124e537-f915-4b7f-ad3e-79186cdd8380" (UID: "d124e537-f915-4b7f-ad3e-79186cdd8380"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:36:38.762192 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.762161    2580 generic.go:356] "Generic (PLEG): container finished" podID="d124e537-f915-4b7f-ad3e-79186cdd8380" containerID="4bfe977bc6d198881a164924709944898004cfa0158c4f76f26170ca92aa1520" exitCode=0
Sep 10 14:36:38.762393 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.762204    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"
Sep 10 14:36:38.762393 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.762251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm" event={"ID":"d124e537-f915-4b7f-ad3e-79186cdd8380","Type":"ContainerDied","Data":"4bfe977bc6d198881a164924709944898004cfa0158c4f76f26170ca92aa1520"}
Sep 10 14:36:38.762393 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.762302    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm" event={"ID":"d124e537-f915-4b7f-ad3e-79186cdd8380","Type":"ContainerDied","Data":"162b68f4aca76e74e04beea190eb12ff2986c34b539cfb0919ec07e4af3d724f"}
Sep 10 14:36:38.762393 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.762325    2580 scope.go:117] "RemoveContainer" containerID="4bfe977bc6d198881a164924709944898004cfa0158c4f76f26170ca92aa1520"
Sep 10 14:36:38.771963 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.771946    2580 scope.go:117] "RemoveContainer" containerID="4bfe977bc6d198881a164924709944898004cfa0158c4f76f26170ca92aa1520"
Sep 10 14:36:38.772282 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:36:38.772258    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bfe977bc6d198881a164924709944898004cfa0158c4f76f26170ca92aa1520\": container with ID starting with 4bfe977bc6d198881a164924709944898004cfa0158c4f76f26170ca92aa1520 not found: ID does not exist" containerID="4bfe977bc6d198881a164924709944898004cfa0158c4f76f26170ca92aa1520"
Sep 10 14:36:38.772370 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.772289    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bfe977bc6d198881a164924709944898004cfa0158c4f76f26170ca92aa1520"} err="failed to get container status \"4bfe977bc6d198881a164924709944898004cfa0158c4f76f26170ca92aa1520\": rpc error: code = NotFound desc = could not find container \"4bfe977bc6d198881a164924709944898004cfa0158c4f76f26170ca92aa1520\": container with ID starting with 4bfe977bc6d198881a164924709944898004cfa0158c4f76f26170ca92aa1520 not found: ID does not exist"
Sep 10 14:36:38.785549 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.785523    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"]
Sep 10 14:36:38.791306 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.791282    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-651c34/noten-model-651c34-kserve-f4fc86d59-sx7cm"]
Sep 10 14:36:38.808811 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.808787    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xqzn\" (UniqueName: \"kubernetes.io/projected/d124e537-f915-4b7f-ad3e-79186cdd8380-kube-api-access-5xqzn\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:38.808811 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.808810    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:38.809001 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.808821    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d124e537-f915-4b7f-ad3e-79186cdd8380-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:38.809001 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:38.808830    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d124e537-f915-4b7f-ad3e-79186cdd8380-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:40.310909 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:40.310868    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d124e537-f915-4b7f-ad3e-79186cdd8380" path="/var/lib/kubelet/pods/d124e537-f915-4b7f-ad3e-79186cdd8380/volumes"
Sep 10 14:36:50.606313 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:50.606279    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"]
Sep 10 14:36:50.606810 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:50.606519    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n" podUID="c8c37460-c57f-4fb2-801d-a559b111d236" containerName="maas-api" containerID="cri-o://d44be8f93c10512ad42749a49a58f3ab99152f3d4e5feaf73deff028a7736094" gracePeriod=30
Sep 10 14:36:51.051744 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.051716    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:36:51.122265 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.122227    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c37460-c57f-4fb2-801d-a559b111d236-openshift-service-ca\") pod \"c8c37460-c57f-4fb2-801d-a559b111d236\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") "
Sep 10 14:36:51.122438 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.122295    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8c37460-c57f-4fb2-801d-a559b111d236-metrics-certs\") pod \"c8c37460-c57f-4fb2-801d-a559b111d236\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") "
Sep 10 14:36:51.122438 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.122343    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmpx5\" (UniqueName: \"kubernetes.io/projected/c8c37460-c57f-4fb2-801d-a559b111d236-kube-api-access-cmpx5\") pod \"c8c37460-c57f-4fb2-801d-a559b111d236\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") "
Sep 10 14:36:51.122438 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.122365    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c37460-c57f-4fb2-801d-a559b111d236-maas-api-tls\") pod \"c8c37460-c57f-4fb2-801d-a559b111d236\" (UID: \"c8c37460-c57f-4fb2-801d-a559b111d236\") "
Sep 10 14:36:51.122870 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.122605    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c37460-c57f-4fb2-801d-a559b111d236-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8c37460-c57f-4fb2-801d-a559b111d236" (UID: "c8c37460-c57f-4fb2-801d-a559b111d236"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:36:51.124513 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.124484    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c37460-c57f-4fb2-801d-a559b111d236-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c8c37460-c57f-4fb2-801d-a559b111d236" (UID: "c8c37460-c57f-4fb2-801d-a559b111d236"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:36:51.124513 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.124499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c37460-c57f-4fb2-801d-a559b111d236-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8c37460-c57f-4fb2-801d-a559b111d236" (UID: "c8c37460-c57f-4fb2-801d-a559b111d236"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:36:51.124674 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.124554    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c37460-c57f-4fb2-801d-a559b111d236-kube-api-access-cmpx5" (OuterVolumeSpecName: "kube-api-access-cmpx5") pod "c8c37460-c57f-4fb2-801d-a559b111d236" (UID: "c8c37460-c57f-4fb2-801d-a559b111d236"). InnerVolumeSpecName "kube-api-access-cmpx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:36:51.223427 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.223328    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8c37460-c57f-4fb2-801d-a559b111d236-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:51.223427 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.223370    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmpx5\" (UniqueName: \"kubernetes.io/projected/c8c37460-c57f-4fb2-801d-a559b111d236-kube-api-access-cmpx5\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:51.223427 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.223381    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c37460-c57f-4fb2-801d-a559b111d236-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:51.223427 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.223391    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c37460-c57f-4fb2-801d-a559b111d236-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:36:51.841830 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.841798    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8c37460-c57f-4fb2-801d-a559b111d236" containerID="d44be8f93c10512ad42749a49a58f3ab99152f3d4e5feaf73deff028a7736094" exitCode=0
Sep 10 14:36:51.842291 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.841840    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"
Sep 10 14:36:51.842291 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.841841    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n" event={"ID":"c8c37460-c57f-4fb2-801d-a559b111d236","Type":"ContainerDied","Data":"d44be8f93c10512ad42749a49a58f3ab99152f3d4e5feaf73deff028a7736094"}
Sep 10 14:36:51.842291 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.841901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n" event={"ID":"c8c37460-c57f-4fb2-801d-a559b111d236","Type":"ContainerDied","Data":"e1e1031986f0b0e9cbdf1c667869ad09e48f6a959b6f40bb2605de8c06fead77"}
Sep 10 14:36:51.842291 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.841928    2580 scope.go:117] "RemoveContainer" containerID="d44be8f93c10512ad42749a49a58f3ab99152f3d4e5feaf73deff028a7736094"
Sep 10 14:36:51.851624 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.851606    2580 scope.go:117] "RemoveContainer" containerID="d44be8f93c10512ad42749a49a58f3ab99152f3d4e5feaf73deff028a7736094"
Sep 10 14:36:51.851913 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:36:51.851895    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44be8f93c10512ad42749a49a58f3ab99152f3d4e5feaf73deff028a7736094\": container with ID starting with d44be8f93c10512ad42749a49a58f3ab99152f3d4e5feaf73deff028a7736094 not found: ID does not exist" containerID="d44be8f93c10512ad42749a49a58f3ab99152f3d4e5feaf73deff028a7736094"
Sep 10 14:36:51.851956 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.851922    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44be8f93c10512ad42749a49a58f3ab99152f3d4e5feaf73deff028a7736094"} err="failed to get container status \"d44be8f93c10512ad42749a49a58f3ab99152f3d4e5feaf73deff028a7736094\": rpc error: code = NotFound desc = could not find container \"d44be8f93c10512ad42749a49a58f3ab99152f3d4e5feaf73deff028a7736094\": container with ID starting with d44be8f93c10512ad42749a49a58f3ab99152f3d4e5feaf73deff028a7736094 not found: ID does not exist"
Sep 10 14:36:51.862700 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.862677    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"]
Sep 10 14:36:51.866640 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:51.866614    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8ee208-94d67cb4d-2924n"]
Sep 10 14:36:52.310868 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:52.310835    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8c37460-c57f-4fb2-801d-a559b111d236" path="/var/lib/kubelet/pods/c8c37460-c57f-4fb2-801d-a559b111d236/volumes"
Sep 10 14:36:53.817116 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817082    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"]
Sep 10 14:36:53.817663 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817644    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98aad721-07cf-460e-b480-c72053cc063d" containerName="maas-api"
Sep 10 14:36:53.817712 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817668    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98aad721-07cf-460e-b480-c72053cc063d" containerName="maas-api"
Sep 10 14:36:53.817712 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817684    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8a57a0f-6eb2-4f16-bcab-cc01c96a5640" containerName="payload-pre-processing"
Sep 10 14:36:53.817712 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817693    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8a57a0f-6eb2-4f16-bcab-cc01c96a5640" containerName="payload-pre-processing"
Sep 10 14:36:53.817712 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817707    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ca45038-4f10-4faa-bd93-b80313a25964" containerName="maas-api"
Sep 10 14:36:53.817908 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817715    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ca45038-4f10-4faa-bd93-b80313a25964" containerName="maas-api"
Sep 10 14:36:53.817908 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817740    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d124e537-f915-4b7f-ad3e-79186cdd8380" containerName="main"
Sep 10 14:36:53.817908 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817749    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d124e537-f915-4b7f-ad3e-79186cdd8380" containerName="main"
Sep 10 14:36:53.817908 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817770    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c37460-c57f-4fb2-801d-a559b111d236" containerName="maas-api"
Sep 10 14:36:53.817908 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817778    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c37460-c57f-4fb2-801d-a559b111d236" containerName="maas-api"
Sep 10 14:36:53.817908 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817865    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="98aad721-07cf-460e-b480-c72053cc063d" containerName="maas-api"
Sep 10 14:36:53.817908 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817876    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d124e537-f915-4b7f-ad3e-79186cdd8380" containerName="main"
Sep 10 14:36:53.817908 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817888    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ca45038-4f10-4faa-bd93-b80313a25964" containerName="maas-api"
Sep 10 14:36:53.817908 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817897    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c37460-c57f-4fb2-801d-a559b111d236" containerName="maas-api"
Sep 10 14:36:53.817908 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.817905    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8a57a0f-6eb2-4f16-bcab-cc01c96a5640" containerName="payload-pre-processing"
Sep 10 14:36:53.822765 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.822741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:53.825075 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.825046    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-8ecc1f-serving-cert\""
Sep 10 14:36:53.831685 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.831620    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"]
Sep 10 14:36:53.946922 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.946887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/824244fd-f32d-4c9f-b0e7-321b28eeb26b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:53.947102 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.946941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824244fd-f32d-4c9f-b0e7-321b28eeb26b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:53.947102 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.947040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cq9h\" (UniqueName: \"kubernetes.io/projected/824244fd-f32d-4c9f-b0e7-321b28eeb26b-kube-api-access-5cq9h\") pod \"maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:53.947102 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:53.947080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824244fd-f32d-4c9f-b0e7-321b28eeb26b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:54.047840 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:54.047804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824244fd-f32d-4c9f-b0e7-321b28eeb26b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:54.048080 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:54.047935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cq9h\" (UniqueName: \"kubernetes.io/projected/824244fd-f32d-4c9f-b0e7-321b28eeb26b-kube-api-access-5cq9h\") pod \"maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:54.048080 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:54.048013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824244fd-f32d-4c9f-b0e7-321b28eeb26b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:54.048218 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:54.048120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/824244fd-f32d-4c9f-b0e7-321b28eeb26b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:54.048218 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:36:54.048171    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-serving-cert: secret "maas-api-e2e-shared-a-w2-8ecc1f-serving-cert" not found
Sep 10 14:36:54.048322 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:36:54.048246    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/824244fd-f32d-4c9f-b0e7-321b28eeb26b-maas-api-tls podName:824244fd-f32d-4c9f-b0e7-321b28eeb26b nodeName:}" failed. No retries permitted until 2026-09-10 14:36:54.548225623 +0000 UTC m=+1032.739753802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/824244fd-f32d-4c9f-b0e7-321b28eeb26b-maas-api-tls") pod "maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k" (UID: "824244fd-f32d-4c9f-b0e7-321b28eeb26b") : secret "maas-api-e2e-shared-a-w2-8ecc1f-serving-cert" not found
Sep 10 14:36:54.048420 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:54.048400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824244fd-f32d-4c9f-b0e7-321b28eeb26b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:54.050517 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:54.050499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/824244fd-f32d-4c9f-b0e7-321b28eeb26b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:54.064265 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:54.064239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cq9h\" (UniqueName: \"kubernetes.io/projected/824244fd-f32d-4c9f-b0e7-321b28eeb26b-kube-api-access-5cq9h\") pod \"maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:54.552201 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:54.552164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824244fd-f32d-4c9f-b0e7-321b28eeb26b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:54.554940 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:54.554914    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824244fd-f32d-4c9f-b0e7-321b28eeb26b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:54.736791 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:54.736754    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:55.008848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:55.008821    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"]
Sep 10 14:36:55.011108 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:36:55.011080    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824244fd_f32d_4c9f_b0e7_321b28eeb26b.slice/crio-43ed80da171fa6257bc24b36c114723a71b6abc8baa95079580e30fe6f0a68ff WatchSource:0}: Error finding container 43ed80da171fa6257bc24b36c114723a71b6abc8baa95079580e30fe6f0a68ff: Status 404 returned error can't find the container with id 43ed80da171fa6257bc24b36c114723a71b6abc8baa95079580e30fe6f0a68ff
Sep 10 14:36:55.871922 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:55.871829    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k" event={"ID":"824244fd-f32d-4c9f-b0e7-321b28eeb26b","Type":"ContainerStarted","Data":"4f75bbe7a505fca0f1c9242ff1fad5495f3367a9b42baf4e327f8da1a9898db4"}
Sep 10 14:36:55.871922 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:55.871880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k" event={"ID":"824244fd-f32d-4c9f-b0e7-321b28eeb26b","Type":"ContainerStarted","Data":"43ed80da171fa6257bc24b36c114723a71b6abc8baa95079580e30fe6f0a68ff"}
Sep 10 14:36:55.872131 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:55.871921    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:36:55.890694 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:36:55.890634    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k" podStartSLOduration=2.446752306 podStartE2EDuration="2.890616789s" totalImagesPullingTime="443.864483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:36:55.888530782 +0000 UTC m=+1034.080058983" watchObservedRunningTime="2026-09-10 14:36:55.890616789 +0000 UTC m=+1034.082144988"
Sep 10 14:37:01.882821 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:01.882793    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:37:02.593830 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.593789    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"]
Sep 10 14:37:02.597726 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.597706    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:02.600095 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.600071    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-7bf16f-serving-cert\""
Sep 10 14:37:02.611218 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.611190    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"]
Sep 10 14:37:02.728993 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.728932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0865ec46-f028-4d9f-a97d-a60e37a487e8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:02.729166 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.729029    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtzpp\" (UniqueName: \"kubernetes.io/projected/0865ec46-f028-4d9f-a97d-a60e37a487e8-kube-api-access-dtzpp\") pod \"maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:02.729166 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.729090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0865ec46-f028-4d9f-a97d-a60e37a487e8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:02.729166 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.729114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0865ec46-f028-4d9f-a97d-a60e37a487e8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:02.829823 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.829786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0865ec46-f028-4d9f-a97d-a60e37a487e8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:02.830031 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.829827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0865ec46-f028-4d9f-a97d-a60e37a487e8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:02.830031 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.829877    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0865ec46-f028-4d9f-a97d-a60e37a487e8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:02.830031 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.829917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtzpp\" (UniqueName: \"kubernetes.io/projected/0865ec46-f028-4d9f-a97d-a60e37a487e8-kube-api-access-dtzpp\") pod \"maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:02.830626 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.830600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0865ec46-f028-4d9f-a97d-a60e37a487e8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:02.832289 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.832262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0865ec46-f028-4d9f-a97d-a60e37a487e8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:02.832408 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.832387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0865ec46-f028-4d9f-a97d-a60e37a487e8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:02.841813 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.841785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtzpp\" (UniqueName: \"kubernetes.io/projected/0865ec46-f028-4d9f-a97d-a60e37a487e8-kube-api-access-dtzpp\") pod \"maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:02.909405 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:02.909375    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:03.182231 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:03.182207    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"]
Sep 10 14:37:03.183894 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:37:03.183865    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0865ec46_f028_4d9f_a97d_a60e37a487e8.slice/crio-c5c6e10ba829127c6a76415d51fc250f36c86f222de7bfff9f15851318315e97 WatchSource:0}: Error finding container c5c6e10ba829127c6a76415d51fc250f36c86f222de7bfff9f15851318315e97: Status 404 returned error can't find the container with id c5c6e10ba829127c6a76415d51fc250f36c86f222de7bfff9f15851318315e97
Sep 10 14:37:03.925537 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:03.925436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f" event={"ID":"0865ec46-f028-4d9f-a97d-a60e37a487e8","Type":"ContainerStarted","Data":"b3086d32cf79e78e18852ffcaec1e07514878e79e36e53036f4ef6dddd9b4573"}
Sep 10 14:37:03.925537 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:03.925498    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:03.925537 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:03.925518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f" event={"ID":"0865ec46-f028-4d9f-a97d-a60e37a487e8","Type":"ContainerStarted","Data":"c5c6e10ba829127c6a76415d51fc250f36c86f222de7bfff9f15851318315e97"}
Sep 10 14:37:03.946191 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:03.946131    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f" podStartSLOduration=1.493334017 podStartE2EDuration="1.94611238s" totalImagesPullingTime="452.778363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:37:03.945889231 +0000 UTC m=+1042.137417430" watchObservedRunningTime="2026-09-10 14:37:03.94611238 +0000 UTC m=+1042.137640583"
Sep 10 14:37:09.936690 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:09.936656    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:37:22.897026 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:22.896967    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"]
Sep 10 14:37:22.900752 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:22.900729    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:22.904832 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:22.904810    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-fc06d4-serving-cert\""
Sep 10 14:37:22.913118 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:22.913093    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"]
Sep 10 14:37:23.018718 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:23.018685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a473c6ab-d2c9-411f-a3b7-03ec0962caac-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:23.018917 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:23.018736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tqpn\" (UniqueName: \"kubernetes.io/projected/a473c6ab-d2c9-411f-a3b7-03ec0962caac-kube-api-access-9tqpn\") pod \"maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:23.018917 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:23.018863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a473c6ab-d2c9-411f-a3b7-03ec0962caac-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:23.018917 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:23.018911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a473c6ab-d2c9-411f-a3b7-03ec0962caac-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:23.119522 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:23.119483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tqpn\" (UniqueName: \"kubernetes.io/projected/a473c6ab-d2c9-411f-a3b7-03ec0962caac-kube-api-access-9tqpn\") pod \"maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:23.119744 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:23.119573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a473c6ab-d2c9-411f-a3b7-03ec0962caac-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:23.119744 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:23.119608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a473c6ab-d2c9-411f-a3b7-03ec0962caac-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:23.119744 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:23.119651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a473c6ab-d2c9-411f-a3b7-03ec0962caac-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:23.119901 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:23.119799    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-serving-cert: secret "maas-api-e2e-shared-b-w2-fc06d4-serving-cert" not found
Sep 10 14:37:23.119901 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:23.119872    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a473c6ab-d2c9-411f-a3b7-03ec0962caac-maas-api-tls podName:a473c6ab-d2c9-411f-a3b7-03ec0962caac nodeName:}" failed. No retries permitted until 2026-09-10 14:37:23.619851166 +0000 UTC m=+1061.811379357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a473c6ab-d2c9-411f-a3b7-03ec0962caac-maas-api-tls") pod "maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8" (UID: "a473c6ab-d2c9-411f-a3b7-03ec0962caac") : secret "maas-api-e2e-shared-b-w2-fc06d4-serving-cert" not found
Sep 10 14:37:23.120484 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:23.120460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a473c6ab-d2c9-411f-a3b7-03ec0962caac-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:23.122472 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:23.122444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a473c6ab-d2c9-411f-a3b7-03ec0962caac-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:23.130344 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:23.130322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tqpn\" (UniqueName: \"kubernetes.io/projected/a473c6ab-d2c9-411f-a3b7-03ec0962caac-kube-api-access-9tqpn\") pod \"maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:23.625190 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:23.625149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a473c6ab-d2c9-411f-a3b7-03ec0962caac-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:23.627898 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:23.627867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a473c6ab-d2c9-411f-a3b7-03ec0962caac-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:23.814235 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:23.814201    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:24.091295 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:24.091269    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"]
Sep 10 14:37:24.092523 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:37:24.092502    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda473c6ab_d2c9_411f_a3b7_03ec0962caac.slice/crio-56d3baeee50796f6aa0cb0a16cab4e1b0a1d68f9b9dc398aa18b3b53654c26cc WatchSource:0}: Error finding container 56d3baeee50796f6aa0cb0a16cab4e1b0a1d68f9b9dc398aa18b3b53654c26cc: Status 404 returned error can't find the container with id 56d3baeee50796f6aa0cb0a16cab4e1b0a1d68f9b9dc398aa18b3b53654c26cc
Sep 10 14:37:25.043870 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:25.043836    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8" event={"ID":"a473c6ab-d2c9-411f-a3b7-03ec0962caac","Type":"ContainerStarted","Data":"dc65e44c745884656abfc16736f4f019dcfa86092d54e4a71d2f15fca78a4581"}
Sep 10 14:37:25.044075 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:25.043918    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8" event={"ID":"a473c6ab-d2c9-411f-a3b7-03ec0962caac","Type":"ContainerStarted","Data":"56d3baeee50796f6aa0cb0a16cab4e1b0a1d68f9b9dc398aa18b3b53654c26cc"}
Sep 10 14:37:25.044075 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:25.043958    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:25.061344 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:25.061283    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8" podStartSLOduration=2.640140463 podStartE2EDuration="3.061262941s" totalImagesPullingTime="421.122478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:37:25.060310552 +0000 UTC m=+1063.251838753" watchObservedRunningTime="2026-09-10 14:37:25.061262941 +0000 UTC m=+1063.252791140"
Sep 10 14:37:30.916278 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:30.916246    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"]
Sep 10 14:37:30.920526 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:30.920503    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:30.922906 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:30.922883    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-59dc59-serving-cert\""
Sep 10 14:37:30.931724 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:30.931696    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"]
Sep 10 14:37:30.988602 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:30.988553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85029c90-a894-4382-8620-4847ad3d1409-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:30.988771 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:30.988705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85029c90-a894-4382-8620-4847ad3d1409-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:30.988771 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:30.988750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm9gr\" (UniqueName: \"kubernetes.io/projected/85029c90-a894-4382-8620-4847ad3d1409-kube-api-access-mm9gr\") pod \"maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:30.988848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:30.988816    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85029c90-a894-4382-8620-4847ad3d1409-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:31.055406 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:31.055379    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:37:31.090249 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:31.090220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85029c90-a894-4382-8620-4847ad3d1409-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:31.090419 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:31.090265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm9gr\" (UniqueName: \"kubernetes.io/projected/85029c90-a894-4382-8620-4847ad3d1409-kube-api-access-mm9gr\") pod \"maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:31.090419 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:31.090300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85029c90-a894-4382-8620-4847ad3d1409-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:31.090419 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:31.090333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85029c90-a894-4382-8620-4847ad3d1409-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:31.090419 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:31.090389    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-serving-cert: secret "maas-api-e2e-shared-b-w3-59dc59-serving-cert" not found
Sep 10 14:37:31.090631 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:31.090466    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85029c90-a894-4382-8620-4847ad3d1409-maas-api-tls podName:85029c90-a894-4382-8620-4847ad3d1409 nodeName:}" failed. No retries permitted until 2026-09-10 14:37:31.59044539 +0000 UTC m=+1069.781973569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85029c90-a894-4382-8620-4847ad3d1409-maas-api-tls") pod "maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h" (UID: "85029c90-a894-4382-8620-4847ad3d1409") : secret "maas-api-e2e-shared-b-w3-59dc59-serving-cert" not found
Sep 10 14:37:31.091086 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:31.091062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85029c90-a894-4382-8620-4847ad3d1409-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:31.092867 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:31.092842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85029c90-a894-4382-8620-4847ad3d1409-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:31.102728 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:31.102703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm9gr\" (UniqueName: \"kubernetes.io/projected/85029c90-a894-4382-8620-4847ad3d1409-kube-api-access-mm9gr\") pod \"maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:31.595625 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:31.595588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85029c90-a894-4382-8620-4847ad3d1409-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:31.598132 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:31.598104    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85029c90-a894-4382-8620-4847ad3d1409-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:31.833513 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:31.833477    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:32.110612 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:32.110573    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"]
Sep 10 14:37:32.113287 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:37:32.113260    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85029c90_a894_4382_8620_4847ad3d1409.slice/crio-b01713405aefd5859e863ea5082d5aed493529fd8bb9ada7a75c0a86efb51a0e WatchSource:0}: Error finding container b01713405aefd5859e863ea5082d5aed493529fd8bb9ada7a75c0a86efb51a0e: Status 404 returned error can't find the container with id b01713405aefd5859e863ea5082d5aed493529fd8bb9ada7a75c0a86efb51a0e
Sep 10 14:37:33.093795 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:33.093756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h" event={"ID":"85029c90-a894-4382-8620-4847ad3d1409","Type":"ContainerStarted","Data":"9fdb31465fe12483a54b38db07c6b33f76e1885604cb67b904897dd6a274df1a"}
Sep 10 14:37:33.094019 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:33.093824    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:33.094019 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:33.093843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h" event={"ID":"85029c90-a894-4382-8620-4847ad3d1409","Type":"ContainerStarted","Data":"b01713405aefd5859e863ea5082d5aed493529fd8bb9ada7a75c0a86efb51a0e"}
Sep 10 14:37:33.109649 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:33.109602    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h" podStartSLOduration=2.739136689 podStartE2EDuration="3.109583321s" totalImagesPullingTime="370.446632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:37:33.109188792 +0000 UTC m=+1071.300716991" watchObservedRunningTime="2026-09-10 14:37:33.109583321 +0000 UTC m=+1071.301111519"
Sep 10 14:37:39.104275 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:39.104186    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:37:47.413069 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:47.413030    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7931-98b2a4"]
Sep 10 14:37:47.417645 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:47.417626    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-98b2a4"
Sep 10 14:37:47.420370 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:47.420347    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qjcz8\""
Sep 10 14:37:47.425051 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:47.425031    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7931-98b2a4"]
Sep 10 14:37:47.452593 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:47.452554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2nsz\" (UniqueName: \"kubernetes.io/projected/41281c32-13c6-4608-a090-c6e93e729e5a-kube-api-access-p2nsz\") pod \"test-curl-7931-98b2a4\" (UID: \"41281c32-13c6-4608-a090-c6e93e729e5a\") " pod="opendatahub/test-curl-7931-98b2a4"
Sep 10 14:37:47.553916 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:47.553879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2nsz\" (UniqueName: \"kubernetes.io/projected/41281c32-13c6-4608-a090-c6e93e729e5a-kube-api-access-p2nsz\") pod \"test-curl-7931-98b2a4\" (UID: \"41281c32-13c6-4608-a090-c6e93e729e5a\") " pod="opendatahub/test-curl-7931-98b2a4"
Sep 10 14:37:47.562230 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:47.562205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2nsz\" (UniqueName: \"kubernetes.io/projected/41281c32-13c6-4608-a090-c6e93e729e5a-kube-api-access-p2nsz\") pod \"test-curl-7931-98b2a4\" (UID: \"41281c32-13c6-4608-a090-c6e93e729e5a\") " pod="opendatahub/test-curl-7931-98b2a4"
Sep 10 14:37:47.727647 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:47.727545    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-98b2a4"
Sep 10 14:37:47.997487 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:47.997461    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7931-98b2a4"]
Sep 10 14:37:47.999588 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:37:47.999561    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41281c32_13c6_4608_a090_c6e93e729e5a.slice/crio-e9d7f9eb30dcc611b5633029868c57ca12649892a313b78dd8be5e4184d68bca WatchSource:0}: Error finding container e9d7f9eb30dcc611b5633029868c57ca12649892a313b78dd8be5e4184d68bca: Status 404 returned error can't find the container with id e9d7f9eb30dcc611b5633029868c57ca12649892a313b78dd8be5e4184d68bca
Sep 10 14:37:48.186953 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:48.186912    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7931-98b2a4" event={"ID":"41281c32-13c6-4608-a090-c6e93e729e5a","Type":"ContainerStarted","Data":"6f416684932524aa288310a3983f504f2735e68c8d1e8cd5703202134530c9e9"}
Sep 10 14:37:48.187136 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:48.186961    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7931-98b2a4" event={"ID":"41281c32-13c6-4608-a090-c6e93e729e5a","Type":"ContainerStarted","Data":"e9d7f9eb30dcc611b5633029868c57ca12649892a313b78dd8be5e4184d68bca"}
Sep 10 14:37:48.203179 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:48.203122    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7931-98b2a4" podStartSLOduration=1.20310568 podStartE2EDuration="1.20310568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:37:48.200473815 +0000 UTC m=+1086.392002011" watchObservedRunningTime="2026-09-10 14:37:48.20310568 +0000 UTC m=+1086.394633877"
Sep 10 14:37:48.780361 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:48.780326    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7931-98b2a4"]
Sep 10 14:37:48.784658 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:48.784629    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7931-98b2a4"]
Sep 10 14:37:48.894885 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:48.894842    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7931-ca2847"]
Sep 10 14:37:48.895351 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:48.895332    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41281c32-13c6-4608-a090-c6e93e729e5a" containerName="test-curl-7931-98b2a4"
Sep 10 14:37:48.895463 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:48.895353    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="41281c32-13c6-4608-a090-c6e93e729e5a" containerName="test-curl-7931-98b2a4"
Sep 10 14:37:48.895463 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:48.895451    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="41281c32-13c6-4608-a090-c6e93e729e5a" containerName="test-curl-7931-98b2a4"
Sep 10 14:37:48.898747 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:48.898726    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-ca2847"
Sep 10 14:37:48.906514 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:48.906381    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7931-ca2847"]
Sep 10 14:37:48.965132 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:48.965095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99jqk\" (UniqueName: \"kubernetes.io/projected/88c5e6a4-b537-43ff-b1db-625ba0fd0555-kube-api-access-99jqk\") pod \"test-curl-7931-ca2847\" (UID: \"88c5e6a4-b537-43ff-b1db-625ba0fd0555\") " pod="opendatahub/test-curl-7931-ca2847"
Sep 10 14:37:49.065909 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:49.065816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99jqk\" (UniqueName: \"kubernetes.io/projected/88c5e6a4-b537-43ff-b1db-625ba0fd0555-kube-api-access-99jqk\") pod \"test-curl-7931-ca2847\" (UID: \"88c5e6a4-b537-43ff-b1db-625ba0fd0555\") " pod="opendatahub/test-curl-7931-ca2847"
Sep 10 14:37:49.074110 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:49.074088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99jqk\" (UniqueName: \"kubernetes.io/projected/88c5e6a4-b537-43ff-b1db-625ba0fd0555-kube-api-access-99jqk\") pod \"test-curl-7931-ca2847\" (UID: \"88c5e6a4-b537-43ff-b1db-625ba0fd0555\") " pod="opendatahub/test-curl-7931-ca2847"
Sep 10 14:37:49.192994 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:49.192943    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7931-98b2a4" container="test-curl-7931-98b2a4"
Sep 10 14:37:49.209066 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:49.209043    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-ca2847"
Sep 10 14:37:49.478023 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:49.477998    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7931-ca2847"]
Sep 10 14:37:49.479782 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:37:49.479754    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88c5e6a4_b537_43ff_b1db_625ba0fd0555.slice/crio-2dfe71d7feada3944d2f76341619f409b6cb058f35fa795ab6f4778711d9d734 WatchSource:0}: Error finding container 2dfe71d7feada3944d2f76341619f409b6cb058f35fa795ab6f4778711d9d734: Status 404 returned error can't find the container with id 2dfe71d7feada3944d2f76341619f409b6cb058f35fa795ab6f4778711d9d734
Sep 10 14:37:50.201221 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:50.201183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7931-ca2847" event={"ID":"88c5e6a4-b537-43ff-b1db-625ba0fd0555","Type":"ContainerStarted","Data":"9128216d77c8a1b1e4c31af44dfe61ed92fc370410aba4a8be02a767d871c3ae"}
Sep 10 14:37:50.201221 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:50.201226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7931-ca2847" event={"ID":"88c5e6a4-b537-43ff-b1db-625ba0fd0555","Type":"ContainerStarted","Data":"2dfe71d7feada3944d2f76341619f409b6cb058f35fa795ab6f4778711d9d734"}
Sep 10 14:37:50.201794 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:50.201229    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7931-98b2a4" podUID="41281c32-13c6-4608-a090-c6e93e729e5a" containerName="test-curl-7931-98b2a4" containerID="cri-o://6f416684932524aa288310a3983f504f2735e68c8d1e8cd5703202134530c9e9" gracePeriod=2
Sep 10 14:37:50.216429 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:50.216383    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7931-ca2847" podStartSLOduration=2.216368198 podStartE2EDuration="2.216368198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:37:50.215698435 +0000 UTC m=+1088.407226634" watchObservedRunningTime="2026-09-10 14:37:50.216368198 +0000 UTC m=+1088.407896393"
Sep 10 14:37:50.217742 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:50.217709    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-98b2a4\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="41281c32-13c6-4608-a090-c6e93e729e5a" pod="opendatahub/test-curl-7931-98b2a4"
Sep 10 14:37:50.219637 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:50.219615    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-98b2a4\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="41281c32-13c6-4608-a090-c6e93e729e5a" pod="opendatahub/test-curl-7931-98b2a4"
Sep 10 14:37:50.811345 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:50.811303    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7931-ca2847"]
Sep 10 14:37:50.818395 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:50.818366    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7931-ca2847"]
Sep 10 14:37:50.935891 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:50.935857    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7931-da16e1"]
Sep 10 14:37:50.936286 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:50.936271    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c5e6a4-b537-43ff-b1db-625ba0fd0555" containerName="test-curl-7931-ca2847"
Sep 10 14:37:50.936332 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:50.936288    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c5e6a4-b537-43ff-b1db-625ba0fd0555" containerName="test-curl-7931-ca2847"
Sep 10 14:37:50.936367 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:50.936353    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c5e6a4-b537-43ff-b1db-625ba0fd0555" containerName="test-curl-7931-ca2847"
Sep 10 14:37:50.939581 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:50.939565    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-da16e1"
Sep 10 14:37:50.945723 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:50.945507    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7931-da16e1"]
Sep 10 14:37:50.980486 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:50.980452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmcbx\" (UniqueName: \"kubernetes.io/projected/76922204-f48a-4d02-afae-f4d20e66aa3c-kube-api-access-xmcbx\") pod \"test-curl-7931-da16e1\" (UID: \"76922204-f48a-4d02-afae-f4d20e66aa3c\") " pod="opendatahub/test-curl-7931-da16e1"
Sep 10 14:37:51.081339 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:51.081302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmcbx\" (UniqueName: \"kubernetes.io/projected/76922204-f48a-4d02-afae-f4d20e66aa3c-kube-api-access-xmcbx\") pod \"test-curl-7931-da16e1\" (UID: \"76922204-f48a-4d02-afae-f4d20e66aa3c\") " pod="opendatahub/test-curl-7931-da16e1"
Sep 10 14:37:51.089440 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:51.089414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmcbx\" (UniqueName: \"kubernetes.io/projected/76922204-f48a-4d02-afae-f4d20e66aa3c-kube-api-access-xmcbx\") pod \"test-curl-7931-da16e1\" (UID: \"76922204-f48a-4d02-afae-f4d20e66aa3c\") " pod="opendatahub/test-curl-7931-da16e1"
Sep 10 14:37:51.212915 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:51.212836    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7931-ca2847" container="test-curl-7931-ca2847"
Sep 10 14:37:51.250919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:51.250887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-da16e1"
Sep 10 14:37:51.523045 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:51.523013    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7931-da16e1"]
Sep 10 14:37:51.524466 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:37:51.524438    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76922204_f48a_4d02_afae_f4d20e66aa3c.slice/crio-c3abac7e63862a576e5cd40059f86a828073dbbdea57e560a2283eb8b57ec437 WatchSource:0}: Error finding container c3abac7e63862a576e5cd40059f86a828073dbbdea57e560a2283eb8b57ec437: Status 404 returned error can't find the container with id c3abac7e63862a576e5cd40059f86a828073dbbdea57e560a2283eb8b57ec437
Sep 10 14:37:52.223518 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.223482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7931-da16e1" event={"ID":"76922204-f48a-4d02-afae-f4d20e66aa3c","Type":"ContainerStarted","Data":"7e7427390fe5ddbf1ebbe2ac8790a15d16de548465c13fe0a627d3d5bddcd817"}
Sep 10 14:37:52.223869 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.223537    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7931-da16e1" event={"ID":"76922204-f48a-4d02-afae-f4d20e66aa3c","Type":"ContainerStarted","Data":"c3abac7e63862a576e5cd40059f86a828073dbbdea57e560a2283eb8b57ec437"}
Sep 10 14:37:52.223869 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.223559    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7931-ca2847" podUID="88c5e6a4-b537-43ff-b1db-625ba0fd0555" containerName="test-curl-7931-ca2847" containerID="cri-o://9128216d77c8a1b1e4c31af44dfe61ed92fc370410aba4a8be02a767d871c3ae" gracePeriod=2
Sep 10 14:37:52.225805 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:52.225780    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-ca2847\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="88c5e6a4-b537-43ff-b1db-625ba0fd0555" pod="opendatahub/test-curl-7931-ca2847"
Sep 10 14:37:52.243150 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:52.243121    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-ca2847\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="88c5e6a4-b537-43ff-b1db-625ba0fd0555" pod="opendatahub/test-curl-7931-ca2847"
Sep 10 14:37:52.243503 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.243465    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7931-da16e1" podStartSLOduration=2.243450423 podStartE2EDuration="2.243450423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:37:52.241022442 +0000 UTC m=+1090.432550642" watchObservedRunningTime="2026-09-10 14:37:52.243450423 +0000 UTC m=+1090.434978662"
Sep 10 14:37:52.309856 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:52.309820    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-ca2847\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="88c5e6a4-b537-43ff-b1db-625ba0fd0555" pod="opendatahub/test-curl-7931-ca2847"
Sep 10 14:37:52.394631 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.394609    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-98b2a4"
Sep 10 14:37:52.396785 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:52.396757    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-98b2a4\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="41281c32-13c6-4608-a090-c6e93e729e5a" pod="opendatahub/test-curl-7931-98b2a4"
Sep 10 14:37:52.492595 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.492509    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2nsz\" (UniqueName: \"kubernetes.io/projected/41281c32-13c6-4608-a090-c6e93e729e5a-kube-api-access-p2nsz\") pod \"41281c32-13c6-4608-a090-c6e93e729e5a\" (UID: \"41281c32-13c6-4608-a090-c6e93e729e5a\") "
Sep 10 14:37:52.494529 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.494503    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41281c32-13c6-4608-a090-c6e93e729e5a-kube-api-access-p2nsz" (OuterVolumeSpecName: "kube-api-access-p2nsz") pod "41281c32-13c6-4608-a090-c6e93e729e5a" (UID: "41281c32-13c6-4608-a090-c6e93e729e5a"). InnerVolumeSpecName "kube-api-access-p2nsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:37:52.593189 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.593156    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2nsz\" (UniqueName: \"kubernetes.io/projected/41281c32-13c6-4608-a090-c6e93e729e5a-kube-api-access-p2nsz\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:37:52.805241 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.805147    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7931-da16e1"]
Sep 10 14:37:52.809435 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.809400    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7931-da16e1"]
Sep 10 14:37:52.927787 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.927755    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7931-bb6a2d"]
Sep 10 14:37:52.928201 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.928186    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76922204-f48a-4d02-afae-f4d20e66aa3c" containerName="test-curl-7931-da16e1"
Sep 10 14:37:52.928258 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.928203    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="76922204-f48a-4d02-afae-f4d20e66aa3c" containerName="test-curl-7931-da16e1"
Sep 10 14:37:52.928298 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.928265    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="76922204-f48a-4d02-afae-f4d20e66aa3c" containerName="test-curl-7931-da16e1"
Sep 10 14:37:52.931377 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.931362    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-bb6a2d"
Sep 10 14:37:52.933783 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:52.933750    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-98b2a4\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="41281c32-13c6-4608-a090-c6e93e729e5a" pod="opendatahub/test-curl-7931-98b2a4"
Sep 10 14:37:52.935512 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.935487    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7931-bb6a2d"]
Sep 10 14:37:52.996264 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:52.996225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssfr8\" (UniqueName: \"kubernetes.io/projected/c337918a-01bc-430d-9b8d-6b267f5bac78-kube-api-access-ssfr8\") pod \"test-curl-7931-bb6a2d\" (UID: \"c337918a-01bc-430d-9b8d-6b267f5bac78\") " pod="opendatahub/test-curl-7931-bb6a2d"
Sep 10 14:37:53.096763 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:53.096680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssfr8\" (UniqueName: \"kubernetes.io/projected/c337918a-01bc-430d-9b8d-6b267f5bac78-kube-api-access-ssfr8\") pod \"test-curl-7931-bb6a2d\" (UID: \"c337918a-01bc-430d-9b8d-6b267f5bac78\") " pod="opendatahub/test-curl-7931-bb6a2d"
Sep 10 14:37:53.105102 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:53.105076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssfr8\" (UniqueName: \"kubernetes.io/projected/c337918a-01bc-430d-9b8d-6b267f5bac78-kube-api-access-ssfr8\") pod \"test-curl-7931-bb6a2d\" (UID: \"c337918a-01bc-430d-9b8d-6b267f5bac78\") " pod="opendatahub/test-curl-7931-bb6a2d"
Sep 10 14:37:53.230173 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:53.230137    2580 generic.go:356] "Generic (PLEG): container finished" podID="41281c32-13c6-4608-a090-c6e93e729e5a" containerID="6f416684932524aa288310a3983f504f2735e68c8d1e8cd5703202134530c9e9" exitCode=137
Sep 10 14:37:53.230606 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:53.230183    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-98b2a4"
Sep 10 14:37:53.230606 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:53.230239    2580 scope.go:117] "RemoveContainer" containerID="6f416684932524aa288310a3983f504f2735e68c8d1e8cd5703202134530c9e9"
Sep 10 14:37:53.230606 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:53.230565    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7931-da16e1" container="test-curl-7931-da16e1"
Sep 10 14:37:53.232578 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:53.232551    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-98b2a4\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="41281c32-13c6-4608-a090-c6e93e729e5a" pod="opendatahub/test-curl-7931-98b2a4"
Sep 10 14:37:53.239476 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:53.239460    2580 scope.go:117] "RemoveContainer" containerID="6f416684932524aa288310a3983f504f2735e68c8d1e8cd5703202134530c9e9"
Sep 10 14:37:53.239734 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:53.239715    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f416684932524aa288310a3983f504f2735e68c8d1e8cd5703202134530c9e9\": container with ID starting with 6f416684932524aa288310a3983f504f2735e68c8d1e8cd5703202134530c9e9 not found: ID does not exist" containerID="6f416684932524aa288310a3983f504f2735e68c8d1e8cd5703202134530c9e9"
Sep 10 14:37:53.239789 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:53.239743    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f416684932524aa288310a3983f504f2735e68c8d1e8cd5703202134530c9e9"} err="failed to get container status \"6f416684932524aa288310a3983f504f2735e68c8d1e8cd5703202134530c9e9\": rpc error: code = NotFound desc = could not find container \"6f416684932524aa288310a3983f504f2735e68c8d1e8cd5703202134530c9e9\": container with ID starting with 6f416684932524aa288310a3983f504f2735e68c8d1e8cd5703202134530c9e9 not found: ID does not exist"
Sep 10 14:37:53.241320 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:53.241294    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-98b2a4\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="41281c32-13c6-4608-a090-c6e93e729e5a" pod="opendatahub/test-curl-7931-98b2a4"
Sep 10 14:37:53.242414 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:53.242401    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-bb6a2d"
Sep 10 14:37:53.511151 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:53.511126    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7931-bb6a2d"]
Sep 10 14:37:53.512722 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:37:53.512694    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc337918a_01bc_430d_9b8d_6b267f5bac78.slice/crio-ac6b2fab4246a4ba6bfa9457a462bc4661f5571025cf5b09796d3ae476735e57 WatchSource:0}: Error finding container ac6b2fab4246a4ba6bfa9457a462bc4661f5571025cf5b09796d3ae476735e57: Status 404 returned error can't find the container with id ac6b2fab4246a4ba6bfa9457a462bc4661f5571025cf5b09796d3ae476735e57
Sep 10 14:37:54.238506 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:54.238470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7931-bb6a2d" event={"ID":"c337918a-01bc-430d-9b8d-6b267f5bac78","Type":"ContainerStarted","Data":"a69199e63df6c46192b2851a3c11e4b3bfa0eae2612267cd567eddd66f710c7a"}
Sep 10 14:37:54.238961 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:54.238518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7931-bb6a2d" event={"ID":"c337918a-01bc-430d-9b8d-6b267f5bac78","Type":"ContainerStarted","Data":"ac6b2fab4246a4ba6bfa9457a462bc4661f5571025cf5b09796d3ae476735e57"}
Sep 10 14:37:54.238961 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:54.238562    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7931-da16e1" podUID="76922204-f48a-4d02-afae-f4d20e66aa3c" containerName="test-curl-7931-da16e1" containerID="cri-o://7e7427390fe5ddbf1ebbe2ac8790a15d16de548465c13fe0a627d3d5bddcd817" gracePeriod=2
Sep 10 14:37:54.241104 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:54.241073    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-da16e1\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="76922204-f48a-4d02-afae-f4d20e66aa3c" pod="opendatahub/test-curl-7931-da16e1"
Sep 10 14:37:54.242943 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:54.242917    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-98b2a4\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="41281c32-13c6-4608-a090-c6e93e729e5a" pod="opendatahub/test-curl-7931-98b2a4"
Sep 10 14:37:54.256279 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:54.256235    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7931-bb6a2d" podStartSLOduration=2.256219728 podStartE2EDuration="2.256219728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:37:54.255678124 +0000 UTC m=+1092.447206322" watchObservedRunningTime="2026-09-10 14:37:54.256219728 +0000 UTC m=+1092.447747926"
Sep 10 14:37:54.257753 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:54.257724    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-da16e1\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="76922204-f48a-4d02-afae-f4d20e66aa3c" pod="opendatahub/test-curl-7931-da16e1"
Sep 10 14:37:54.259741 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:54.259718    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-98b2a4\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="41281c32-13c6-4608-a090-c6e93e729e5a" pod="opendatahub/test-curl-7931-98b2a4"
Sep 10 14:37:54.311220 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:54.311189    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41281c32-13c6-4608-a090-c6e93e729e5a" path="/var/lib/kubelet/pods/41281c32-13c6-4608-a090-c6e93e729e5a/volumes"
Sep 10 14:37:54.399549 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:54.399527    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-ca2847"
Sep 10 14:37:54.405784 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:54.405762    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99jqk\" (UniqueName: \"kubernetes.io/projected/88c5e6a4-b537-43ff-b1db-625ba0fd0555-kube-api-access-99jqk\") pod \"88c5e6a4-b537-43ff-b1db-625ba0fd0555\" (UID: \"88c5e6a4-b537-43ff-b1db-625ba0fd0555\") "
Sep 10 14:37:54.407887 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:54.407862    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c5e6a4-b537-43ff-b1db-625ba0fd0555-kube-api-access-99jqk" (OuterVolumeSpecName: "kube-api-access-99jqk") pod "88c5e6a4-b537-43ff-b1db-625ba0fd0555" (UID: "88c5e6a4-b537-43ff-b1db-625ba0fd0555"). InnerVolumeSpecName "kube-api-access-99jqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:37:54.507130 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:54.507033    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99jqk\" (UniqueName: \"kubernetes.io/projected/88c5e6a4-b537-43ff-b1db-625ba0fd0555-kube-api-access-99jqk\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:37:54.807341 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:54.807260    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7931-bb6a2d"]
Sep 10 14:37:54.810952 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:54.810927    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7931-bb6a2d"]
Sep 10 14:37:55.244713 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:55.244676    2580 generic.go:356] "Generic (PLEG): container finished" podID="88c5e6a4-b537-43ff-b1db-625ba0fd0555" containerID="9128216d77c8a1b1e4c31af44dfe61ed92fc370410aba4a8be02a767d871c3ae" exitCode=137
Sep 10 14:37:55.245130 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:55.244726    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-ca2847"
Sep 10 14:37:55.245130 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:55.244782    2580 scope.go:117] "RemoveContainer" containerID="9128216d77c8a1b1e4c31af44dfe61ed92fc370410aba4a8be02a767d871c3ae"
Sep 10 14:37:55.245130 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:55.245090    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7931-bb6a2d" container="test-curl-7931-bb6a2d"
Sep 10 14:37:55.253790 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:55.253771    2580 scope.go:117] "RemoveContainer" containerID="9128216d77c8a1b1e4c31af44dfe61ed92fc370410aba4a8be02a767d871c3ae"
Sep 10 14:37:55.254126 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:55.254106    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9128216d77c8a1b1e4c31af44dfe61ed92fc370410aba4a8be02a767d871c3ae\": container with ID starting with 9128216d77c8a1b1e4c31af44dfe61ed92fc370410aba4a8be02a767d871c3ae not found: ID does not exist" containerID="9128216d77c8a1b1e4c31af44dfe61ed92fc370410aba4a8be02a767d871c3ae"
Sep 10 14:37:55.254192 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:55.254134    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9128216d77c8a1b1e4c31af44dfe61ed92fc370410aba4a8be02a767d871c3ae"} err="failed to get container status \"9128216d77c8a1b1e4c31af44dfe61ed92fc370410aba4a8be02a767d871c3ae\": rpc error: code = NotFound desc = could not find container \"9128216d77c8a1b1e4c31af44dfe61ed92fc370410aba4a8be02a767d871c3ae\": container with ID starting with 9128216d77c8a1b1e4c31af44dfe61ed92fc370410aba4a8be02a767d871c3ae not found: ID does not exist"
Sep 10 14:37:55.255178 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:55.255159    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-ca2847\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="88c5e6a4-b537-43ff-b1db-625ba0fd0555" pod="opendatahub/test-curl-7931-ca2847"
Sep 10 14:37:56.251768 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:56.251735    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7931-bb6a2d" podUID="c337918a-01bc-430d-9b8d-6b267f5bac78" containerName="test-curl-7931-bb6a2d" containerID="cri-o://a69199e63df6c46192b2851a3c11e4b3bfa0eae2612267cd567eddd66f710c7a" gracePeriod=2
Sep 10 14:37:56.254342 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:56.254311    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-bb6a2d\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="c337918a-01bc-430d-9b8d-6b267f5bac78" pod="opendatahub/test-curl-7931-bb6a2d"
Sep 10 14:37:56.256273 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:56.256247    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-ca2847\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="88c5e6a4-b537-43ff-b1db-625ba0fd0555" pod="opendatahub/test-curl-7931-ca2847"
Sep 10 14:37:56.310868 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:56.310837    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88c5e6a4-b537-43ff-b1db-625ba0fd0555" path="/var/lib/kubelet/pods/88c5e6a4-b537-43ff-b1db-625ba0fd0555/volumes"
Sep 10 14:37:56.415755 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:56.415732    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-da16e1"
Sep 10 14:37:56.419621 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:56.419603    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmcbx\" (UniqueName: \"kubernetes.io/projected/76922204-f48a-4d02-afae-f4d20e66aa3c-kube-api-access-xmcbx\") pod \"76922204-f48a-4d02-afae-f4d20e66aa3c\" (UID: \"76922204-f48a-4d02-afae-f4d20e66aa3c\") "
Sep 10 14:37:56.421531 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:56.421500    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76922204-f48a-4d02-afae-f4d20e66aa3c-kube-api-access-xmcbx" (OuterVolumeSpecName: "kube-api-access-xmcbx") pod "76922204-f48a-4d02-afae-f4d20e66aa3c" (UID: "76922204-f48a-4d02-afae-f4d20e66aa3c"). InnerVolumeSpecName "kube-api-access-xmcbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:37:56.521012 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:56.520916    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmcbx\" (UniqueName: \"kubernetes.io/projected/76922204-f48a-4d02-afae-f4d20e66aa3c-kube-api-access-xmcbx\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:37:57.258576 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:57.258539    2580 generic.go:356] "Generic (PLEG): container finished" podID="76922204-f48a-4d02-afae-f4d20e66aa3c" containerID="7e7427390fe5ddbf1ebbe2ac8790a15d16de548465c13fe0a627d3d5bddcd817" exitCode=137
Sep 10 14:37:57.258576 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:57.258572    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-da16e1"
Sep 10 14:37:57.259088 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:57.258614    2580 scope.go:117] "RemoveContainer" containerID="7e7427390fe5ddbf1ebbe2ac8790a15d16de548465c13fe0a627d3d5bddcd817"
Sep 10 14:37:57.267722 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:57.267706    2580 scope.go:117] "RemoveContainer" containerID="7e7427390fe5ddbf1ebbe2ac8790a15d16de548465c13fe0a627d3d5bddcd817"
Sep 10 14:37:57.267994 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:57.267957    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7427390fe5ddbf1ebbe2ac8790a15d16de548465c13fe0a627d3d5bddcd817\": container with ID starting with 7e7427390fe5ddbf1ebbe2ac8790a15d16de548465c13fe0a627d3d5bddcd817 not found: ID does not exist" containerID="7e7427390fe5ddbf1ebbe2ac8790a15d16de548465c13fe0a627d3d5bddcd817"
Sep 10 14:37:57.268058 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:57.268000    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7427390fe5ddbf1ebbe2ac8790a15d16de548465c13fe0a627d3d5bddcd817"} err="failed to get container status \"7e7427390fe5ddbf1ebbe2ac8790a15d16de548465c13fe0a627d3d5bddcd817\": rpc error: code = NotFound desc = could not find container \"7e7427390fe5ddbf1ebbe2ac8790a15d16de548465c13fe0a627d3d5bddcd817\": container with ID starting with 7e7427390fe5ddbf1ebbe2ac8790a15d16de548465c13fe0a627d3d5bddcd817 not found: ID does not exist"
Sep 10 14:37:57.269752 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:57.269731    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-da16e1\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="76922204-f48a-4d02-afae-f4d20e66aa3c" pod="opendatahub/test-curl-7931-da16e1"
Sep 10 14:37:58.310286 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:58.310249    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76922204-f48a-4d02-afae-f4d20e66aa3c" path="/var/lib/kubelet/pods/76922204-f48a-4d02-afae-f4d20e66aa3c/volumes"
Sep 10 14:37:58.442561 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:58.442537    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-bb6a2d"
Sep 10 14:37:58.536841 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:58.536754    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssfr8\" (UniqueName: \"kubernetes.io/projected/c337918a-01bc-430d-9b8d-6b267f5bac78-kube-api-access-ssfr8\") pod \"c337918a-01bc-430d-9b8d-6b267f5bac78\" (UID: \"c337918a-01bc-430d-9b8d-6b267f5bac78\") "
Sep 10 14:37:58.538811 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:58.538790    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c337918a-01bc-430d-9b8d-6b267f5bac78-kube-api-access-ssfr8" (OuterVolumeSpecName: "kube-api-access-ssfr8") pod "c337918a-01bc-430d-9b8d-6b267f5bac78" (UID: "c337918a-01bc-430d-9b8d-6b267f5bac78"). InnerVolumeSpecName "kube-api-access-ssfr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:37:58.637439 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:58.637391    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssfr8\" (UniqueName: \"kubernetes.io/projected/c337918a-01bc-430d-9b8d-6b267f5bac78-kube-api-access-ssfr8\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:37:59.272392 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.272360    2580 generic.go:356] "Generic (PLEG): container finished" podID="c337918a-01bc-430d-9b8d-6b267f5bac78" containerID="a69199e63df6c46192b2851a3c11e4b3bfa0eae2612267cd567eddd66f710c7a" exitCode=137
Sep 10 14:37:59.272560 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.272405    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7931-bb6a2d"
Sep 10 14:37:59.272560 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.272459    2580 scope.go:117] "RemoveContainer" containerID="a69199e63df6c46192b2851a3c11e4b3bfa0eae2612267cd567eddd66f710c7a"
Sep 10 14:37:59.281426 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.281409    2580 scope.go:117] "RemoveContainer" containerID="a69199e63df6c46192b2851a3c11e4b3bfa0eae2612267cd567eddd66f710c7a"
Sep 10 14:37:59.281684 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:59.281664    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69199e63df6c46192b2851a3c11e4b3bfa0eae2612267cd567eddd66f710c7a\": container with ID starting with a69199e63df6c46192b2851a3c11e4b3bfa0eae2612267cd567eddd66f710c7a not found: ID does not exist" containerID="a69199e63df6c46192b2851a3c11e4b3bfa0eae2612267cd567eddd66f710c7a"
Sep 10 14:37:59.281727 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.281691    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69199e63df6c46192b2851a3c11e4b3bfa0eae2612267cd567eddd66f710c7a"} err="failed to get container status \"a69199e63df6c46192b2851a3c11e4b3bfa0eae2612267cd567eddd66f710c7a\": rpc error: code = NotFound desc = could not find container \"a69199e63df6c46192b2851a3c11e4b3bfa0eae2612267cd567eddd66f710c7a\": container with ID starting with a69199e63df6c46192b2851a3c11e4b3bfa0eae2612267cd567eddd66f710c7a not found: ID does not exist"
Sep 10 14:37:59.282530 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:59.282511    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-bb6a2d\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="c337918a-01bc-430d-9b8d-6b267f5bac78" pod="opendatahub/test-curl-7931-bb6a2d"
Sep 10 14:37:59.919738 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.919708    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"]
Sep 10 14:37:59.920763 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.920737    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c337918a-01bc-430d-9b8d-6b267f5bac78" containerName="test-curl-7931-bb6a2d"
Sep 10 14:37:59.920763 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.920763    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c337918a-01bc-430d-9b8d-6b267f5bac78" containerName="test-curl-7931-bb6a2d"
Sep 10 14:37:59.920910 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.920882    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c337918a-01bc-430d-9b8d-6b267f5bac78" containerName="test-curl-7931-bb6a2d"
Sep 10 14:37:59.928659 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.928635    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:37:59.931438 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.931395    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-25792506-serving-cert\""
Sep 10 14:37:59.934197 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.934167    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"]
Sep 10 14:37:59.946240 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:37:59.946185    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7931-bb6a2d\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="c337918a-01bc-430d-9b8d-6b267f5bac78" pod="opendatahub/test-curl-7931-bb6a2d"
Sep 10 14:37:59.946869 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.946849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9906b10-94e4-4549-8b0a-7def13afecdf-maas-api-tls\") pod \"maas-api-e2e-mt-25792506-ddc478857-rscs2\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:37:59.947006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.946887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9906b10-94e4-4549-8b0a-7def13afecdf-openshift-service-ca\") pod \"maas-api-e2e-mt-25792506-ddc478857-rscs2\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:37:59.947006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.946912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9906b10-94e4-4549-8b0a-7def13afecdf-metrics-certs\") pod \"maas-api-e2e-mt-25792506-ddc478857-rscs2\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:37:59.947115 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:37:59.947050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgv5x\" (UniqueName: \"kubernetes.io/projected/f9906b10-94e4-4549-8b0a-7def13afecdf-kube-api-access-mgv5x\") pod \"maas-api-e2e-mt-25792506-ddc478857-rscs2\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:38:00.047794 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:00.047762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9906b10-94e4-4549-8b0a-7def13afecdf-openshift-service-ca\") pod \"maas-api-e2e-mt-25792506-ddc478857-rscs2\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:38:00.047962 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:00.047815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9906b10-94e4-4549-8b0a-7def13afecdf-metrics-certs\") pod \"maas-api-e2e-mt-25792506-ddc478857-rscs2\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:38:00.047962 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:00.047870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgv5x\" (UniqueName: \"kubernetes.io/projected/f9906b10-94e4-4549-8b0a-7def13afecdf-kube-api-access-mgv5x\") pod \"maas-api-e2e-mt-25792506-ddc478857-rscs2\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:38:00.048131 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:00.048030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9906b10-94e4-4549-8b0a-7def13afecdf-maas-api-tls\") pod \"maas-api-e2e-mt-25792506-ddc478857-rscs2\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:38:00.048519 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:00.048493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9906b10-94e4-4549-8b0a-7def13afecdf-openshift-service-ca\") pod \"maas-api-e2e-mt-25792506-ddc478857-rscs2\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:38:00.050528 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:00.050496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9906b10-94e4-4549-8b0a-7def13afecdf-maas-api-tls\") pod \"maas-api-e2e-mt-25792506-ddc478857-rscs2\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:38:00.050640 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:00.050499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9906b10-94e4-4549-8b0a-7def13afecdf-metrics-certs\") pod \"maas-api-e2e-mt-25792506-ddc478857-rscs2\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:38:00.057169 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:00.057143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgv5x\" (UniqueName: \"kubernetes.io/projected/f9906b10-94e4-4549-8b0a-7def13afecdf-kube-api-access-mgv5x\") pod \"maas-api-e2e-mt-25792506-ddc478857-rscs2\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:38:00.241518 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:00.241419    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:38:00.322106 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:00.322067    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c337918a-01bc-430d-9b8d-6b267f5bac78" path="/var/lib/kubelet/pods/c337918a-01bc-430d-9b8d-6b267f5bac78/volumes"
Sep 10 14:38:00.520918 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:00.520889    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"]
Sep 10 14:38:00.522268 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:38:00.522241    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9906b10_94e4_4549_8b0a_7def13afecdf.slice/crio-d051c2a485dbd49301768482631e8e942b502cb9f17c880d7856b4b0707fe91c WatchSource:0}: Error finding container d051c2a485dbd49301768482631e8e942b502cb9f17c880d7856b4b0707fe91c: Status 404 returned error can't find the container with id d051c2a485dbd49301768482631e8e942b502cb9f17c880d7856b4b0707fe91c
Sep 10 14:38:01.287605 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:01.287566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2" event={"ID":"f9906b10-94e4-4549-8b0a-7def13afecdf","Type":"ContainerStarted","Data":"26ab1f0090bbb896bd187c179678290fddf29b1e91c1c6913a53655b82801bea"}
Sep 10 14:38:01.288062 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:01.287628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2" event={"ID":"f9906b10-94e4-4549-8b0a-7def13afecdf","Type":"ContainerStarted","Data":"d051c2a485dbd49301768482631e8e942b502cb9f17c880d7856b4b0707fe91c"}
Sep 10 14:38:01.288062 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:01.287665    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:38:01.305818 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:01.305763    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2" podStartSLOduration=1.657373483 podStartE2EDuration="2.305746005s" totalImagesPullingTime="648.372522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:38:01.303235155 +0000 UTC m=+1099.494763356" watchObservedRunningTime="2026-09-10 14:38:01.305746005 +0000 UTC m=+1099.497274206"
Sep 10 14:38:07.298780 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:07.298746    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:38:11.011403 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.011371    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"]
Sep 10 14:38:11.014711 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.014691    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.018457 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.018431    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7bf16f\"/\"default-dockercfg-7kps9\""
Sep 10 14:38:11.018612 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.018448    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7bf16f\"/\"kube-root-ca.crt\""
Sep 10 14:38:11.018612 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.018478    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7bf16f\"/\"openshift-service-ca.crt\""
Sep 10 14:38:11.018612 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.018450    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7bf16f\"/\"auth-test-model-7bf16f-kserve-self-signed-certs\""
Sep 10 14:38:11.025703 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.025678    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"]
Sep 10 14:38:11.045506 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.045469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e730fe69-4db2-46e8-b7c9-9c9f0794731f-tls-certs\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.045506 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.045501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-dshm\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.045716 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.045592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-model-cache\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.045716 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.045631    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwc2s\" (UniqueName: \"kubernetes.io/projected/e730fe69-4db2-46e8-b7c9-9c9f0794731f-kube-api-access-dwc2s\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.045716 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.045657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-home\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.045716 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.045689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-tmp-dir\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.147043 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.147011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-dshm\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.147231 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.147095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-model-cache\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.147231 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.147126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwc2s\" (UniqueName: \"kubernetes.io/projected/e730fe69-4db2-46e8-b7c9-9c9f0794731f-kube-api-access-dwc2s\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.147231 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.147150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-home\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.147395 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.147281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-tmp-dir\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.147395 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.147357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e730fe69-4db2-46e8-b7c9-9c9f0794731f-tls-certs\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.147561 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.147543    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-home\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.147628 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.147581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-model-cache\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.147676 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.147625    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-tmp-dir\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.149390 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.149370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-dshm\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.149621 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.149605    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e730fe69-4db2-46e8-b7c9-9c9f0794731f-tls-certs\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.155418 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.155397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwc2s\" (UniqueName: \"kubernetes.io/projected/e730fe69-4db2-46e8-b7c9-9c9f0794731f-kube-api-access-dwc2s\") pod \"auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.325390 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.325294    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:11.606359 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:11.606331    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"]
Sep 10 14:38:11.607889 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:38:11.607861    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode730fe69_4db2_46e8_b7c9_9c9f0794731f.slice/crio-c072af7dfe613a1745069ffd45fe8edae9eafe83c5252125af1ef701bc7cbc08 WatchSource:0}: Error finding container c072af7dfe613a1745069ffd45fe8edae9eafe83c5252125af1ef701bc7cbc08: Status 404 returned error can't find the container with id c072af7dfe613a1745069ffd45fe8edae9eafe83c5252125af1ef701bc7cbc08
Sep 10 14:38:12.352684 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:12.352646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb" event={"ID":"e730fe69-4db2-46e8-b7c9-9c9f0794731f","Type":"ContainerStarted","Data":"cef5254fd1c52b9b2b9586788308c023ed19e2dc055d022ad7b8220f32a1f9e7"}
Sep 10 14:38:12.353094 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:12.352693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb" event={"ID":"e730fe69-4db2-46e8-b7c9-9c9f0794731f","Type":"ContainerStarted","Data":"c072af7dfe613a1745069ffd45fe8edae9eafe83c5252125af1ef701bc7cbc08"}
Sep 10 14:38:12.370755 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:12.370698    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb" podStartSLOduration=2.370683974 podStartE2EDuration="2.370683974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:38:12.368478526 +0000 UTC m=+1110.560006723" watchObservedRunningTime="2026-09-10 14:38:12.370683974 +0000 UTC m=+1110.562212172"
Sep 10 14:38:21.325622 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:21.325579    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:21.326123 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:21.325646    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:21.338308 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:21.338285    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:21.414033 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:21.414002    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:38:41.514477 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.514445    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"]
Sep 10 14:38:41.517930 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.517911    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.520461 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.520430    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-25792506\"/\"default-dockercfg-n9g2x\""
Sep 10 14:38:41.520602 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.520468    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-25792506\"/\"kube-root-ca.crt\""
Sep 10 14:38:41.521452 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.521434    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-25792506\"/\"e2e-lifecycle-model-25792506-kserve-self-signed-certs\""
Sep 10 14:38:41.521555 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.521486    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-25792506\"/\"openshift-service-ca.crt\""
Sep 10 14:38:41.529388 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.529364    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"]
Sep 10 14:38:41.547686 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.547655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-dshm\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.547855 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.547691    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-tls-certs\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.547855 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.547764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-home\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.547855 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.547809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk98l\" (UniqueName: \"kubernetes.io/projected/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-kube-api-access-jk98l\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.547855 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.547843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-tmp-dir\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.548134 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.547873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-model-cache\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.648933 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.648895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-tls-certs\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.649130 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.648949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-home\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.649130 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.648997    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk98l\" (UniqueName: \"kubernetes.io/projected/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-kube-api-access-jk98l\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.649130 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.649019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-tmp-dir\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.649130 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.649042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-model-cache\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.649130 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.649118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-dshm\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.649428 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.649402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-home\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.649472 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.649415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-model-cache\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.649516 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.649481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-tmp-dir\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.651206 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.651185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-dshm\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.651487 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.651470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-tls-certs\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.657197 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.657171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk98l\" (UniqueName: \"kubernetes.io/projected/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-kube-api-access-jk98l\") pod \"e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") " pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:41.828883 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:41.828800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:42.102129 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:42.102103    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"]
Sep 10 14:38:42.103808 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:38:42.103781    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2b6d80b_119f_4eb2_8935_a35ad2cfe1c3.slice/crio-af03e1b8346e49a8f60a829e39dbe4bea3f70850eb6f01001b2a1bde7daa511c WatchSource:0}: Error finding container af03e1b8346e49a8f60a829e39dbe4bea3f70850eb6f01001b2a1bde7daa511c: Status 404 returned error can't find the container with id af03e1b8346e49a8f60a829e39dbe4bea3f70850eb6f01001b2a1bde7daa511c
Sep 10 14:38:42.523764 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:42.523728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x" event={"ID":"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3","Type":"ContainerStarted","Data":"01316620a38b3eeaf177f7ec797aa0aeac8a0b302a9784a2fc1d82b251819a15"}
Sep 10 14:38:42.524195 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:42.523772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x" event={"ID":"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3","Type":"ContainerStarted","Data":"af03e1b8346e49a8f60a829e39dbe4bea3f70850eb6f01001b2a1bde7daa511c"}
Sep 10 14:38:42.541833 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:42.541779    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x" podStartSLOduration=1.541764081 podStartE2EDuration="1.541764081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:38:42.538866228 +0000 UTC m=+1140.730394426" watchObservedRunningTime="2026-09-10 14:38:42.541764081 +0000 UTC m=+1140.733292346"
Sep 10 14:38:51.829291 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:51.829251    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:51.829694 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:51.829313    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:51.842101 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:51.842073    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:38:52.593344 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:38:52.593315    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:39:42.337148 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:42.337120    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:39:42.345042 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:42.345018    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:39:42.447660 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:42.447628    2580 scope.go:117] "RemoveContainer" containerID="925bc1b69b232201e5d0a06f9f187dc6fd73d670e007f462023e210223586cb4"
Sep 10 14:39:44.374888 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.374846    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"]
Sep 10 14:39:44.375381 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.375086    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2" podUID="f9906b10-94e4-4549-8b0a-7def13afecdf" containerName="maas-api" containerID="cri-o://26ab1f0090bbb896bd187c179678290fddf29b1e91c1c6913a53655b82801bea" gracePeriod=30
Sep 10 14:39:44.637442 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.637416    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:39:44.764316 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.764281    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9906b10-94e4-4549-8b0a-7def13afecdf-openshift-service-ca\") pod \"f9906b10-94e4-4549-8b0a-7def13afecdf\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") "
Sep 10 14:39:44.764499 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.764359    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9906b10-94e4-4549-8b0a-7def13afecdf-metrics-certs\") pod \"f9906b10-94e4-4549-8b0a-7def13afecdf\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") "
Sep 10 14:39:44.764499 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.764386    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9906b10-94e4-4549-8b0a-7def13afecdf-maas-api-tls\") pod \"f9906b10-94e4-4549-8b0a-7def13afecdf\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") "
Sep 10 14:39:44.764614 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.764562    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgv5x\" (UniqueName: \"kubernetes.io/projected/f9906b10-94e4-4549-8b0a-7def13afecdf-kube-api-access-mgv5x\") pod \"f9906b10-94e4-4549-8b0a-7def13afecdf\" (UID: \"f9906b10-94e4-4549-8b0a-7def13afecdf\") "
Sep 10 14:39:44.764729 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.764704    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9906b10-94e4-4549-8b0a-7def13afecdf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f9906b10-94e4-4549-8b0a-7def13afecdf" (UID: "f9906b10-94e4-4549-8b0a-7def13afecdf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:39:44.764957 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.764934    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9906b10-94e4-4549-8b0a-7def13afecdf-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:39:44.766541 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.766518    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9906b10-94e4-4549-8b0a-7def13afecdf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f9906b10-94e4-4549-8b0a-7def13afecdf" (UID: "f9906b10-94e4-4549-8b0a-7def13afecdf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:39:44.766636 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.766550    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9906b10-94e4-4549-8b0a-7def13afecdf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f9906b10-94e4-4549-8b0a-7def13afecdf" (UID: "f9906b10-94e4-4549-8b0a-7def13afecdf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:39:44.766678 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.766655    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9906b10-94e4-4549-8b0a-7def13afecdf-kube-api-access-mgv5x" (OuterVolumeSpecName: "kube-api-access-mgv5x") pod "f9906b10-94e4-4549-8b0a-7def13afecdf" (UID: "f9906b10-94e4-4549-8b0a-7def13afecdf"). InnerVolumeSpecName "kube-api-access-mgv5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:39:44.865632 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.865592    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgv5x\" (UniqueName: \"kubernetes.io/projected/f9906b10-94e4-4549-8b0a-7def13afecdf-kube-api-access-mgv5x\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:39:44.865632 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.865627    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9906b10-94e4-4549-8b0a-7def13afecdf-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:39:44.865632 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.865636    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9906b10-94e4-4549-8b0a-7def13afecdf-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:39:44.895904 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.895809    2580 generic.go:356] "Generic (PLEG): container finished" podID="f9906b10-94e4-4549-8b0a-7def13afecdf" containerID="26ab1f0090bbb896bd187c179678290fddf29b1e91c1c6913a53655b82801bea" exitCode=0
Sep 10 14:39:44.895904 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.895860    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"
Sep 10 14:39:44.896119 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.895905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2" event={"ID":"f9906b10-94e4-4549-8b0a-7def13afecdf","Type":"ContainerDied","Data":"26ab1f0090bbb896bd187c179678290fddf29b1e91c1c6913a53655b82801bea"}
Sep 10 14:39:44.896119 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.895959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2" event={"ID":"f9906b10-94e4-4549-8b0a-7def13afecdf","Type":"ContainerDied","Data":"d051c2a485dbd49301768482631e8e942b502cb9f17c880d7856b4b0707fe91c"}
Sep 10 14:39:44.896119 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.896015    2580 scope.go:117] "RemoveContainer" containerID="26ab1f0090bbb896bd187c179678290fddf29b1e91c1c6913a53655b82801bea"
Sep 10 14:39:44.906116 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.906099    2580 scope.go:117] "RemoveContainer" containerID="26ab1f0090bbb896bd187c179678290fddf29b1e91c1c6913a53655b82801bea"
Sep 10 14:39:44.906434 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:39:44.906408    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ab1f0090bbb896bd187c179678290fddf29b1e91c1c6913a53655b82801bea\": container with ID starting with 26ab1f0090bbb896bd187c179678290fddf29b1e91c1c6913a53655b82801bea not found: ID does not exist" containerID="26ab1f0090bbb896bd187c179678290fddf29b1e91c1c6913a53655b82801bea"
Sep 10 14:39:44.906534 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.906440    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ab1f0090bbb896bd187c179678290fddf29b1e91c1c6913a53655b82801bea"} err="failed to get container status \"26ab1f0090bbb896bd187c179678290fddf29b1e91c1c6913a53655b82801bea\": rpc error: code = NotFound desc = could not find container \"26ab1f0090bbb896bd187c179678290fddf29b1e91c1c6913a53655b82801bea\": container with ID starting with 26ab1f0090bbb896bd187c179678290fddf29b1e91c1c6913a53655b82801bea not found: ID does not exist"
Sep 10 14:39:44.926684 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.926652    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"]
Sep 10 14:39:44.931801 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:44.931768    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25792506-ddc478857-rscs2"]
Sep 10 14:39:46.311013 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:46.310959    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9906b10-94e4-4549-8b0a-7def13afecdf" path="/var/lib/kubelet/pods/f9906b10-94e4-4549-8b0a-7def13afecdf/volumes"
Sep 10 14:39:52.708195 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:39:52.708157    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"]
Sep 10 14:40:07.728091 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:07.728021    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x" podUID="d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3" containerName="main" containerID="cri-o://01316620a38b3eeaf177f7ec797aa0aeac8a0b302a9784a2fc1d82b251819a15" gracePeriod=45
Sep 10 14:40:07.978254 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:07.978194    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:40:08.029914 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.029879    2580 generic.go:356] "Generic (PLEG): container finished" podID="d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3" containerID="01316620a38b3eeaf177f7ec797aa0aeac8a0b302a9784a2fc1d82b251819a15" exitCode=0
Sep 10 14:40:08.030086 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.029922    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"
Sep 10 14:40:08.030086 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.029953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x" event={"ID":"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3","Type":"ContainerDied","Data":"01316620a38b3eeaf177f7ec797aa0aeac8a0b302a9784a2fc1d82b251819a15"}
Sep 10 14:40:08.030086 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.030013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x" event={"ID":"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3","Type":"ContainerDied","Data":"af03e1b8346e49a8f60a829e39dbe4bea3f70850eb6f01001b2a1bde7daa511c"}
Sep 10 14:40:08.030086 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.030036    2580 scope.go:117] "RemoveContainer" containerID="01316620a38b3eeaf177f7ec797aa0aeac8a0b302a9784a2fc1d82b251819a15"
Sep 10 14:40:08.039485 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.039462    2580 scope.go:117] "RemoveContainer" containerID="01316620a38b3eeaf177f7ec797aa0aeac8a0b302a9784a2fc1d82b251819a15"
Sep 10 14:40:08.039793 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:40:08.039775    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01316620a38b3eeaf177f7ec797aa0aeac8a0b302a9784a2fc1d82b251819a15\": container with ID starting with 01316620a38b3eeaf177f7ec797aa0aeac8a0b302a9784a2fc1d82b251819a15 not found: ID does not exist" containerID="01316620a38b3eeaf177f7ec797aa0aeac8a0b302a9784a2fc1d82b251819a15"
Sep 10 14:40:08.039884 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.039801    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01316620a38b3eeaf177f7ec797aa0aeac8a0b302a9784a2fc1d82b251819a15"} err="failed to get container status \"01316620a38b3eeaf177f7ec797aa0aeac8a0b302a9784a2fc1d82b251819a15\": rpc error: code = NotFound desc = could not find container \"01316620a38b3eeaf177f7ec797aa0aeac8a0b302a9784a2fc1d82b251819a15\": container with ID starting with 01316620a38b3eeaf177f7ec797aa0aeac8a0b302a9784a2fc1d82b251819a15 not found: ID does not exist"
Sep 10 14:40:08.088296 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.088263    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-model-cache\") pod \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") "
Sep 10 14:40:08.088472 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.088315    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-home\") pod \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") "
Sep 10 14:40:08.088472 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.088362    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-dshm\") pod \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") "
Sep 10 14:40:08.088472 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.088390    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk98l\" (UniqueName: \"kubernetes.io/projected/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-kube-api-access-jk98l\") pod \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") "
Sep 10 14:40:08.088472 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.088410    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-tmp-dir\") pod \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") "
Sep 10 14:40:08.088472 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.088471    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-tls-certs\") pod \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\" (UID: \"d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3\") "
Sep 10 14:40:08.088750 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.088575    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3" (UID: "d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:40:08.088750 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.088592    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-home" (OuterVolumeSpecName: "home") pod "d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3" (UID: "d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:40:08.088750 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.088675    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3" (UID: "d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:40:08.088878 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.088807    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:40:08.088878 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.088821    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:40:08.088878 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.088829    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:40:08.090611 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.090585    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-dshm" (OuterVolumeSpecName: "dshm") pod "d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3" (UID: "d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:40:08.090725 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.090624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3" (UID: "d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:40:08.090725 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.090639    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-kube-api-access-jk98l" (OuterVolumeSpecName: "kube-api-access-jk98l") pod "d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3" (UID: "d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3"). InnerVolumeSpecName "kube-api-access-jk98l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:40:08.190313 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.190274    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:40:08.190313 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.190303    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:40:08.190313 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.190314    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk98l\" (UniqueName: \"kubernetes.io/projected/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3-kube-api-access-jk98l\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:40:08.346787 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.346757    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"]
Sep 10 14:40:08.352457 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:08.352428    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-25792506/e2e-lifecycle-model-25792506-kserve-69bf688f5-xdg9x"]
Sep 10 14:40:10.309869 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:40:10.309834    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3" path="/var/lib/kubelet/pods/d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3/volumes"
Sep 10 14:41:01.722222 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:01.722180    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7934-787d92"]
Sep 10 14:41:01.722682 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:01.722601    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3" containerName="main"
Sep 10 14:41:01.722682 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:01.722613    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3" containerName="main"
Sep 10 14:41:01.722682 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:01.722643    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9906b10-94e4-4549-8b0a-7def13afecdf" containerName="maas-api"
Sep 10 14:41:01.722682 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:01.722650    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9906b10-94e4-4549-8b0a-7def13afecdf" containerName="maas-api"
Sep 10 14:41:01.722815 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:01.722713    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9906b10-94e4-4549-8b0a-7def13afecdf" containerName="maas-api"
Sep 10 14:41:01.722815 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:01.722722    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b6d80b-119f-4eb2-8935-a35ad2cfe1c3" containerName="main"
Sep 10 14:41:01.727232 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:01.727211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7934-787d92"
Sep 10 14:41:01.729802 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:01.729776    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qjcz8\""
Sep 10 14:41:01.731647 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:01.731619    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7934-787d92"]
Sep 10 14:41:01.792884 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:01.792844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2qm9\" (UniqueName: \"kubernetes.io/projected/40c3c702-11a3-44e8-9ffa-13036cc57399-kube-api-access-w2qm9\") pod \"test-curl-7934-787d92\" (UID: \"40c3c702-11a3-44e8-9ffa-13036cc57399\") " pod="opendatahub/test-curl-7934-787d92"
Sep 10 14:41:01.894092 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:01.894049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2qm9\" (UniqueName: \"kubernetes.io/projected/40c3c702-11a3-44e8-9ffa-13036cc57399-kube-api-access-w2qm9\") pod \"test-curl-7934-787d92\" (UID: \"40c3c702-11a3-44e8-9ffa-13036cc57399\") " pod="opendatahub/test-curl-7934-787d92"
Sep 10 14:41:01.902639 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:01.902619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2qm9\" (UniqueName: \"kubernetes.io/projected/40c3c702-11a3-44e8-9ffa-13036cc57399-kube-api-access-w2qm9\") pod \"test-curl-7934-787d92\" (UID: \"40c3c702-11a3-44e8-9ffa-13036cc57399\") " pod="opendatahub/test-curl-7934-787d92"
Sep 10 14:41:02.038246 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:02.038157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7934-787d92"
Sep 10 14:41:02.311297 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:02.311269    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7934-787d92"]
Sep 10 14:41:02.311651 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:41:02.311626    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40c3c702_11a3_44e8_9ffa_13036cc57399.slice/crio-dd9ac7ee939ae8c5c8e6d65151244db7ec8ad1ed6c85a602d62c9d27ebe0e573 WatchSource:0}: Error finding container dd9ac7ee939ae8c5c8e6d65151244db7ec8ad1ed6c85a602d62c9d27ebe0e573: Status 404 returned error can't find the container with id dd9ac7ee939ae8c5c8e6d65151244db7ec8ad1ed6c85a602d62c9d27ebe0e573
Sep 10 14:41:02.341328 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:02.341294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7934-787d92" event={"ID":"40c3c702-11a3-44e8-9ffa-13036cc57399","Type":"ContainerStarted","Data":"dd9ac7ee939ae8c5c8e6d65151244db7ec8ad1ed6c85a602d62c9d27ebe0e573"}
Sep 10 14:41:03.348547 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:03.348511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7934-787d92" event={"ID":"40c3c702-11a3-44e8-9ffa-13036cc57399","Type":"ContainerStarted","Data":"a6ca441f5fec39a057ec3fc879468075a48886e7892154d6d00b04ab49dc03c2"}
Sep 10 14:41:03.365475 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:03.365418    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7934-787d92" podStartSLOduration=2.3653999519999998 podStartE2EDuration="2.365399952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:41:03.362554784 +0000 UTC m=+1281.554082982" watchObservedRunningTime="2026-09-10 14:41:03.365399952 +0000 UTC m=+1281.556928149"
Sep 10 14:41:03.940789 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:03.940752    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7934-787d92"]
Sep 10 14:41:03.945630 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:03.945597    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7934-787d92"]
Sep 10 14:41:04.353990 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:41:04.353940    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7934-787d92" container="test-curl-7934-787d92"
Sep 10 14:41:05.296252 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:05.296207    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"]
Sep 10 14:41:05.360016 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:05.359523    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7934-787d92" podUID="40c3c702-11a3-44e8-9ffa-13036cc57399" containerName="test-curl-7934-787d92" containerID="cri-o://a6ca441f5fec39a057ec3fc879468075a48886e7892154d6d00b04ab49dc03c2" gracePeriod=2
Sep 10 14:41:07.529471 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:07.529443    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7934-787d92"
Sep 10 14:41:07.647381 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:07.647352    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2qm9\" (UniqueName: \"kubernetes.io/projected/40c3c702-11a3-44e8-9ffa-13036cc57399-kube-api-access-w2qm9\") pod \"40c3c702-11a3-44e8-9ffa-13036cc57399\" (UID: \"40c3c702-11a3-44e8-9ffa-13036cc57399\") "
Sep 10 14:41:07.649390 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:07.649365    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c3c702-11a3-44e8-9ffa-13036cc57399-kube-api-access-w2qm9" (OuterVolumeSpecName: "kube-api-access-w2qm9") pod "40c3c702-11a3-44e8-9ffa-13036cc57399" (UID: "40c3c702-11a3-44e8-9ffa-13036cc57399"). InnerVolumeSpecName "kube-api-access-w2qm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:41:07.748208 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:07.748172    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2qm9\" (UniqueName: \"kubernetes.io/projected/40c3c702-11a3-44e8-9ffa-13036cc57399-kube-api-access-w2qm9\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:41:08.310037 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:08.309996    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40c3c702-11a3-44e8-9ffa-13036cc57399" path="/var/lib/kubelet/pods/40c3c702-11a3-44e8-9ffa-13036cc57399/volumes"
Sep 10 14:41:08.377112 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:08.377081    2580 generic.go:356] "Generic (PLEG): container finished" podID="40c3c702-11a3-44e8-9ffa-13036cc57399" containerID="a6ca441f5fec39a057ec3fc879468075a48886e7892154d6d00b04ab49dc03c2" exitCode=137
Sep 10 14:41:08.377298 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:08.377130    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7934-787d92"
Sep 10 14:41:08.377298 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:08.377184    2580 scope.go:117] "RemoveContainer" containerID="a6ca441f5fec39a057ec3fc879468075a48886e7892154d6d00b04ab49dc03c2"
Sep 10 14:41:08.385905 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:08.385887    2580 scope.go:117] "RemoveContainer" containerID="a6ca441f5fec39a057ec3fc879468075a48886e7892154d6d00b04ab49dc03c2"
Sep 10 14:41:08.386203 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:41:08.386185    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ca441f5fec39a057ec3fc879468075a48886e7892154d6d00b04ab49dc03c2\": container with ID starting with a6ca441f5fec39a057ec3fc879468075a48886e7892154d6d00b04ab49dc03c2 not found: ID does not exist" containerID="a6ca441f5fec39a057ec3fc879468075a48886e7892154d6d00b04ab49dc03c2"
Sep 10 14:41:08.386273 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:08.386212    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ca441f5fec39a057ec3fc879468075a48886e7892154d6d00b04ab49dc03c2"} err="failed to get container status \"a6ca441f5fec39a057ec3fc879468075a48886e7892154d6d00b04ab49dc03c2\": rpc error: code = NotFound desc = could not find container \"a6ca441f5fec39a057ec3fc879468075a48886e7892154d6d00b04ab49dc03c2\": container with ID starting with a6ca441f5fec39a057ec3fc879468075a48886e7892154d6d00b04ab49dc03c2 not found: ID does not exist"
Sep 10 14:41:18.419471 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.419435    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"]
Sep 10 14:41:18.420011 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.419990    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c3c702-11a3-44e8-9ffa-13036cc57399" containerName="test-curl-7934-787d92"
Sep 10 14:41:18.420011 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.420011    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c3c702-11a3-44e8-9ffa-13036cc57399" containerName="test-curl-7934-787d92"
Sep 10 14:41:18.420155 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.420094    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c3c702-11a3-44e8-9ffa-13036cc57399" containerName="test-curl-7934-787d92"
Sep 10 14:41:18.423130 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.423111    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.426075 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.426055    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7bf16f\"/\"sub-test-model-7bf16f-kserve-self-signed-certs\""
Sep 10 14:41:18.434417 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.434389    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"]
Sep 10 14:41:18.546527 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.546493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-home\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.546719 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.546558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-model-cache\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.546719 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.546578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68a66be9-b80f-4210-b86e-577d581b0209-tls-certs\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.546719 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.546628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-dshm\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.546719 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.546693    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-tmp-dir\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.546877 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.546753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc8nh\" (UniqueName: \"kubernetes.io/projected/68a66be9-b80f-4210-b86e-577d581b0209-kube-api-access-rc8nh\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.647493 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.647457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc8nh\" (UniqueName: \"kubernetes.io/projected/68a66be9-b80f-4210-b86e-577d581b0209-kube-api-access-rc8nh\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.647683 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.647511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-home\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.647683 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.647551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-model-cache\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.647683 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.647565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68a66be9-b80f-4210-b86e-577d581b0209-tls-certs\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.647683 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.647598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-dshm\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.647683 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.647634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-tmp-dir\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.648030 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.648004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-model-cache\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.648030 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.648022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-home\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.648112 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.648082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-tmp-dir\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.649836 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.649807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-dshm\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.650119 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.650100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68a66be9-b80f-4210-b86e-577d581b0209-tls-certs\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.654770 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.654750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc8nh\" (UniqueName: \"kubernetes.io/projected/68a66be9-b80f-4210-b86e-577d581b0209-kube-api-access-rc8nh\") pod \"sub-test-model-7bf16f-kserve-54f9d88995-s52rf\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:18.735278 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:18.735199    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:19.008267 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:19.008238    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"]
Sep 10 14:41:19.010056 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:41:19.010022    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68a66be9_b80f_4210_b86e_577d581b0209.slice/crio-09bcb0d66dff5c2d3afc46a4bdf9614904654147eccc0b231525a93342917107 WatchSource:0}: Error finding container 09bcb0d66dff5c2d3afc46a4bdf9614904654147eccc0b231525a93342917107: Status 404 returned error can't find the container with id 09bcb0d66dff5c2d3afc46a4bdf9614904654147eccc0b231525a93342917107
Sep 10 14:41:19.443755 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:19.443719    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf" event={"ID":"68a66be9-b80f-4210-b86e-577d581b0209","Type":"ContainerStarted","Data":"dd483b65ece6de4360f97a968f26f0221ca1540507fa8fd77ef831c6c9897567"}
Sep 10 14:41:19.444148 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:19.443769    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf" event={"ID":"68a66be9-b80f-4210-b86e-577d581b0209","Type":"ContainerStarted","Data":"09bcb0d66dff5c2d3afc46a4bdf9614904654147eccc0b231525a93342917107"}
Sep 10 14:41:19.460791 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:19.460740    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf" podStartSLOduration=1.460722993 podStartE2EDuration="1.460722993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:41:19.45864336 +0000 UTC m=+1297.650171560" watchObservedRunningTime="2026-09-10 14:41:19.460722993 +0000 UTC m=+1297.652251191"
Sep 10 14:41:20.316078 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.316042    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb" podUID="e730fe69-4db2-46e8-b7c9-9c9f0794731f" containerName="main" containerID="cri-o://cef5254fd1c52b9b2b9586788308c023ed19e2dc055d022ad7b8220f32a1f9e7" gracePeriod=45
Sep 10 14:41:20.451258 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.451231    2580 generic.go:356] "Generic (PLEG): container finished" podID="e730fe69-4db2-46e8-b7c9-9c9f0794731f" containerID="cef5254fd1c52b9b2b9586788308c023ed19e2dc055d022ad7b8220f32a1f9e7" exitCode=0
Sep 10 14:41:20.451592 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.451300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb" event={"ID":"e730fe69-4db2-46e8-b7c9-9c9f0794731f","Type":"ContainerDied","Data":"cef5254fd1c52b9b2b9586788308c023ed19e2dc055d022ad7b8220f32a1f9e7"}
Sep 10 14:41:20.572017 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.571940    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:41:20.669709 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.669674    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-tmp-dir\") pod \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") "
Sep 10 14:41:20.669881 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.669717    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-home\") pod \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") "
Sep 10 14:41:20.669881 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.669751    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e730fe69-4db2-46e8-b7c9-9c9f0794731f-tls-certs\") pod \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") "
Sep 10 14:41:20.669881 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.669792    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwc2s\" (UniqueName: \"kubernetes.io/projected/e730fe69-4db2-46e8-b7c9-9c9f0794731f-kube-api-access-dwc2s\") pod \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") "
Sep 10 14:41:20.670051 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.669934    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-model-cache\") pod \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") "
Sep 10 14:41:20.670106 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.670039    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e730fe69-4db2-46e8-b7c9-9c9f0794731f" (UID: "e730fe69-4db2-46e8-b7c9-9c9f0794731f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:20.670106 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.670059    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-home" (OuterVolumeSpecName: "home") pod "e730fe69-4db2-46e8-b7c9-9c9f0794731f" (UID: "e730fe69-4db2-46e8-b7c9-9c9f0794731f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:20.670106 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.670087    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-dshm\") pod \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\" (UID: \"e730fe69-4db2-46e8-b7c9-9c9f0794731f\") "
Sep 10 14:41:20.670259 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.670155    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e730fe69-4db2-46e8-b7c9-9c9f0794731f" (UID: "e730fe69-4db2-46e8-b7c9-9c9f0794731f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:20.670497 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.670477    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:41:20.670613 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.670520    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:41:20.670613 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.670543    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:41:20.672111 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.672079    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-dshm" (OuterVolumeSpecName: "dshm") pod "e730fe69-4db2-46e8-b7c9-9c9f0794731f" (UID: "e730fe69-4db2-46e8-b7c9-9c9f0794731f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:41:20.672214 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.672151    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e730fe69-4db2-46e8-b7c9-9c9f0794731f-kube-api-access-dwc2s" (OuterVolumeSpecName: "kube-api-access-dwc2s") pod "e730fe69-4db2-46e8-b7c9-9c9f0794731f" (UID: "e730fe69-4db2-46e8-b7c9-9c9f0794731f"). InnerVolumeSpecName "kube-api-access-dwc2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:41:20.672214 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.672165    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e730fe69-4db2-46e8-b7c9-9c9f0794731f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e730fe69-4db2-46e8-b7c9-9c9f0794731f" (UID: "e730fe69-4db2-46e8-b7c9-9c9f0794731f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:41:20.771799 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.771762    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e730fe69-4db2-46e8-b7c9-9c9f0794731f-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:41:20.771799 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.771792    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e730fe69-4db2-46e8-b7c9-9c9f0794731f-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:41:20.771799 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:20.771803    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwc2s\" (UniqueName: \"kubernetes.io/projected/e730fe69-4db2-46e8-b7c9-9c9f0794731f-kube-api-access-dwc2s\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:41:21.458439 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:21.458408    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"
Sep 10 14:41:21.458927 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:21.458461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb" event={"ID":"e730fe69-4db2-46e8-b7c9-9c9f0794731f","Type":"ContainerDied","Data":"c072af7dfe613a1745069ffd45fe8edae9eafe83c5252125af1ef701bc7cbc08"}
Sep 10 14:41:21.458927 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:21.458522    2580 scope.go:117] "RemoveContainer" containerID="cef5254fd1c52b9b2b9586788308c023ed19e2dc055d022ad7b8220f32a1f9e7"
Sep 10 14:41:21.485434 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:21.485407    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"]
Sep 10 14:41:21.490604 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:21.490574    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/auth-test-model-7bf16f-kserve-54f48d5ccf-cnhkb"]
Sep 10 14:41:22.316337 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:22.316297    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e730fe69-4db2-46e8-b7c9-9c9f0794731f" path="/var/lib/kubelet/pods/e730fe69-4db2-46e8-b7c9-9c9f0794731f/volumes"
Sep 10 14:41:28.735407 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:28.735366    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:28.735814 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:28.735429    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:28.747822 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:28.747794    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:29.515359 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:29.515331    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:41:35.302630 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.302589    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"]
Sep 10 14:41:35.303273 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.303257    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e730fe69-4db2-46e8-b7c9-9c9f0794731f" containerName="main"
Sep 10 14:41:35.303321 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.303276    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e730fe69-4db2-46e8-b7c9-9c9f0794731f" containerName="main"
Sep 10 14:41:35.303401 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.303390    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e730fe69-4db2-46e8-b7c9-9c9f0794731f" containerName="main"
Sep 10 14:41:35.308149 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.308127    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:35.311038 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.311013    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1d2af2-serving-cert\""
Sep 10 14:41:35.315771 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.315751    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"]
Sep 10 14:41:35.407095 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.407064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jcr8\" (UniqueName: \"kubernetes.io/projected/7383a8f2-1883-4022-9b81-aec86c647be0-kube-api-access-9jcr8\") pod \"maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:35.407268 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.407117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7383a8f2-1883-4022-9b81-aec86c647be0-metrics-certs\") pod \"maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:35.407268 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.407207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7383a8f2-1883-4022-9b81-aec86c647be0-maas-api-tls\") pod \"maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:35.407268 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.407233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7383a8f2-1883-4022-9b81-aec86c647be0-openshift-service-ca\") pod \"maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:35.508111 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.508071    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7383a8f2-1883-4022-9b81-aec86c647be0-metrics-certs\") pod \"maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:35.508308 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.508155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7383a8f2-1883-4022-9b81-aec86c647be0-maas-api-tls\") pod \"maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:35.508308 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.508185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7383a8f2-1883-4022-9b81-aec86c647be0-openshift-service-ca\") pod \"maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:35.508308 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.508256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jcr8\" (UniqueName: \"kubernetes.io/projected/7383a8f2-1883-4022-9b81-aec86c647be0-kube-api-access-9jcr8\") pod \"maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:35.508308 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:41:35.508294    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-serving-cert: secret "maas-api-e2e-api-a-1d2af2-serving-cert" not found
Sep 10 14:41:35.508535 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:41:35.508372    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7383a8f2-1883-4022-9b81-aec86c647be0-maas-api-tls podName:7383a8f2-1883-4022-9b81-aec86c647be0 nodeName:}" failed. No retries permitted until 2026-09-10 14:41:36.008355887 +0000 UTC m=+1314.199884062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7383a8f2-1883-4022-9b81-aec86c647be0-maas-api-tls") pod "maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh" (UID: "7383a8f2-1883-4022-9b81-aec86c647be0") : secret "maas-api-e2e-api-a-1d2af2-serving-cert" not found
Sep 10 14:41:35.508923 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.508895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7383a8f2-1883-4022-9b81-aec86c647be0-openshift-service-ca\") pod \"maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:35.510586 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.510559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7383a8f2-1883-4022-9b81-aec86c647be0-metrics-certs\") pod \"maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:35.516967 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:35.516939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jcr8\" (UniqueName: \"kubernetes.io/projected/7383a8f2-1883-4022-9b81-aec86c647be0-kube-api-access-9jcr8\") pod \"maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:36.013780 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:36.013744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7383a8f2-1883-4022-9b81-aec86c647be0-maas-api-tls\") pod \"maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:36.016432 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:36.016396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7383a8f2-1883-4022-9b81-aec86c647be0-maas-api-tls\") pod \"maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:36.221709 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:36.221668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:36.499867 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:36.499834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"]
Sep 10 14:41:36.501349 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:41:36.501320    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7383a8f2_1883_4022_9b81_aec86c647be0.slice/crio-7ce87add9a16dda8c462f586e5a33c877173046f841135e78698b4e4e2492e7b WatchSource:0}: Error finding container 7ce87add9a16dda8c462f586e5a33c877173046f841135e78698b4e4e2492e7b: Status 404 returned error can't find the container with id 7ce87add9a16dda8c462f586e5a33c877173046f841135e78698b4e4e2492e7b
Sep 10 14:41:36.502754 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:36.502738    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:41:36.544098 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:36.544008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh" event={"ID":"7383a8f2-1883-4022-9b81-aec86c647be0","Type":"ContainerStarted","Data":"7ce87add9a16dda8c462f586e5a33c877173046f841135e78698b4e4e2492e7b"}
Sep 10 14:41:37.551553 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:37.551519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh" event={"ID":"7383a8f2-1883-4022-9b81-aec86c647be0","Type":"ContainerStarted","Data":"ae2990970e2a00980ff5df50ed4a3e53855fd463d087742ff5765ed24d201348"}
Sep 10 14:41:37.552026 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:37.551630    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:37.573310 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:37.573259    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh" podStartSLOduration=2.152670151 podStartE2EDuration="2.573241803s" totalImagesPullingTime="420.571652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:41:37.571273497 +0000 UTC m=+1315.762801694" watchObservedRunningTime="2026-09-10 14:41:37.573241803 +0000 UTC m=+1315.764770000"
Sep 10 14:41:43.563102 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:43.563072    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:41:45.008817 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.008776    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"]
Sep 10 14:41:45.014225 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.014192    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.018012 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.017965    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-59dc59\"/\"kube-root-ca.crt\""
Sep 10 14:41:45.018377 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.018349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-59dc59\"/\"default-dockercfg-kcnjl\""
Sep 10 14:41:45.018377 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.018378    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-59dc59\"/\"openshift-service-ca.crt\""
Sep 10 14:41:45.018555 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.018454    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-59dc59\"/\"sub-test-model-59dc59-kserve-self-signed-certs\""
Sep 10 14:41:45.022382 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.022088    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"]
Sep 10 14:41:45.103343 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.103313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-home\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.103343 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.103350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-tmp-dir\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.103552 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.103373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5bnb\" (UniqueName: \"kubernetes.io/projected/4d4dccd6-c979-4621-adca-f69a990ffa5e-kube-api-access-q5bnb\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.103552 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.103494    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4dccd6-c979-4621-adca-f69a990ffa5e-tls-certs\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.103552 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.103532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-model-cache\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.103669 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.103564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-dshm\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.204943 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.204903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-dshm\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.205139 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.205006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-home\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.205139 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.205039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-tmp-dir\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.205139 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.205062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5bnb\" (UniqueName: \"kubernetes.io/projected/4d4dccd6-c979-4621-adca-f69a990ffa5e-kube-api-access-q5bnb\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.205139 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.205111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4dccd6-c979-4621-adca-f69a990ffa5e-tls-certs\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.205139 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.205133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-model-cache\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.205466 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.205441    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-home\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.205531 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.205470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-tmp-dir\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.205531 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.205513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-model-cache\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.207253 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.207237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-dshm\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.207517 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.207500    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4dccd6-c979-4621-adca-f69a990ffa5e-tls-certs\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.213332 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.213313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5bnb\" (UniqueName: \"kubernetes.io/projected/4d4dccd6-c979-4621-adca-f69a990ffa5e-kube-api-access-q5bnb\") pod \"sub-test-model-59dc59-kserve-865f8f64b9-nkr4q\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.325581 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.325487    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:45.597589 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:45.597561    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"]
Sep 10 14:41:45.599100 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:41:45.599073    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d4dccd6_c979_4621_adca_f69a990ffa5e.slice/crio-881a6897d698d470c2160ec1e29d95a6047998c007fc3791b9396d27fde73915 WatchSource:0}: Error finding container 881a6897d698d470c2160ec1e29d95a6047998c007fc3791b9396d27fde73915: Status 404 returned error can't find the container with id 881a6897d698d470c2160ec1e29d95a6047998c007fc3791b9396d27fde73915
Sep 10 14:41:46.603154 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:46.603118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q" event={"ID":"4d4dccd6-c979-4621-adca-f69a990ffa5e","Type":"ContainerStarted","Data":"9d196bba007beba6699c12b021d0c0d4ff0b71796f65ac5f118953c76bed2a6c"}
Sep 10 14:41:46.603601 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:46.603162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q" event={"ID":"4d4dccd6-c979-4621-adca-f69a990ffa5e","Type":"ContainerStarted","Data":"881a6897d698d470c2160ec1e29d95a6047998c007fc3791b9396d27fde73915"}
Sep 10 14:41:46.621147 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:46.621093    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q" podStartSLOduration=2.621078154 podStartE2EDuration="2.621078154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:41:46.618689749 +0000 UTC m=+1324.810217951" watchObservedRunningTime="2026-09-10 14:41:46.621078154 +0000 UTC m=+1324.812606352"
Sep 10 14:41:55.326067 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:55.326027    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:55.326067 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:55.326077    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:55.338725 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:55.338696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:41:55.662742 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:41:55.662714    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:42:04.508421 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.508340    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"]
Sep 10 14:42:04.513561 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.513536    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:04.516232 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.516205    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4ba551-serving-cert\""
Sep 10 14:42:04.526880 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.526837    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"]
Sep 10 14:42:04.695698 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.695664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzpsz\" (UniqueName: \"kubernetes.io/projected/141612d7-93f1-437b-b876-fdcba29cc166-kube-api-access-mzpsz\") pod \"maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:04.695888 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.695747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/141612d7-93f1-437b-b876-fdcba29cc166-metrics-certs\") pod \"maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:04.695888 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.695776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141612d7-93f1-437b-b876-fdcba29cc166-openshift-service-ca\") pod \"maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:04.695888 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.695791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141612d7-93f1-437b-b876-fdcba29cc166-maas-api-tls\") pod \"maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:04.796537 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.796454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/141612d7-93f1-437b-b876-fdcba29cc166-metrics-certs\") pod \"maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:04.796537 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.796503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141612d7-93f1-437b-b876-fdcba29cc166-openshift-service-ca\") pod \"maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:04.796537 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.796528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141612d7-93f1-437b-b876-fdcba29cc166-maas-api-tls\") pod \"maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:04.796841 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.796586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzpsz\" (UniqueName: \"kubernetes.io/projected/141612d7-93f1-437b-b876-fdcba29cc166-kube-api-access-mzpsz\") pod \"maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:04.797229 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.797201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141612d7-93f1-437b-b876-fdcba29cc166-openshift-service-ca\") pod \"maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:04.799184 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.799165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141612d7-93f1-437b-b876-fdcba29cc166-maas-api-tls\") pod \"maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:04.799299 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.799197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/141612d7-93f1-437b-b876-fdcba29cc166-metrics-certs\") pod \"maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:04.813413 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.813378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzpsz\" (UniqueName: \"kubernetes.io/projected/141612d7-93f1-437b-b876-fdcba29cc166-kube-api-access-mzpsz\") pod \"maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:04.827086 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:04.827061    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:05.118141 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:05.118109    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"]
Sep 10 14:42:05.118459 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:42:05.118432    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141612d7_93f1_437b_b876_fdcba29cc166.slice/crio-500401ec99534b3c7ed1f789c67aa907f5471ad45e30c6d3c05bcf9fc3616c02 WatchSource:0}: Error finding container 500401ec99534b3c7ed1f789c67aa907f5471ad45e30c6d3c05bcf9fc3616c02: Status 404 returned error can't find the container with id 500401ec99534b3c7ed1f789c67aa907f5471ad45e30c6d3c05bcf9fc3616c02
Sep 10 14:42:05.715194 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:05.715149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn" event={"ID":"141612d7-93f1-437b-b876-fdcba29cc166","Type":"ContainerStarted","Data":"e32025361338ef9a3f7d60b38d8dca5910a3eef237487b7dfa3132c04f488efc"}
Sep 10 14:42:05.715641 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:05.715244    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn" event={"ID":"141612d7-93f1-437b-b876-fdcba29cc166","Type":"ContainerStarted","Data":"500401ec99534b3c7ed1f789c67aa907f5471ad45e30c6d3c05bcf9fc3616c02"}
Sep 10 14:42:05.715641 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:05.715292    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:05.731817 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:05.731756    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn" podStartSLOduration=1.3482418219999999 podStartE2EDuration="1.731739587s" totalImagesPullingTime="383.497765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:05.730924175 +0000 UTC m=+1343.922452373" watchObservedRunningTime="2026-09-10 14:42:05.731739587 +0000 UTC m=+1343.923267834"
Sep 10 14:42:11.727180 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:11.727148    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:42:20.360354 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:20.360319    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7934-531ef0"]
Sep 10 14:42:20.364167 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:20.364147    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7934-531ef0"
Sep 10 14:42:20.367063 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:20.367039    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qjcz8\""
Sep 10 14:42:20.372840 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:20.372554    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7934-531ef0"]
Sep 10 14:42:20.447010 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:20.446958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcgcf\" (UniqueName: \"kubernetes.io/projected/9b2349a1-ab3c-446e-b0cd-3f562c9277ce-kube-api-access-fcgcf\") pod \"test-curl-7934-531ef0\" (UID: \"9b2349a1-ab3c-446e-b0cd-3f562c9277ce\") " pod="opendatahub/test-curl-7934-531ef0"
Sep 10 14:42:20.548477 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:20.548446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcgcf\" (UniqueName: \"kubernetes.io/projected/9b2349a1-ab3c-446e-b0cd-3f562c9277ce-kube-api-access-fcgcf\") pod \"test-curl-7934-531ef0\" (UID: \"9b2349a1-ab3c-446e-b0cd-3f562c9277ce\") " pod="opendatahub/test-curl-7934-531ef0"
Sep 10 14:42:20.556823 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:20.556791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcgcf\" (UniqueName: \"kubernetes.io/projected/9b2349a1-ab3c-446e-b0cd-3f562c9277ce-kube-api-access-fcgcf\") pod \"test-curl-7934-531ef0\" (UID: \"9b2349a1-ab3c-446e-b0cd-3f562c9277ce\") " pod="opendatahub/test-curl-7934-531ef0"
Sep 10 14:42:20.676122 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:20.676081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7934-531ef0"
Sep 10 14:42:20.953182 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:20.953157    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7934-531ef0"]
Sep 10 14:42:20.955158 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:42:20.955129    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b2349a1_ab3c_446e_b0cd_3f562c9277ce.slice/crio-fbb816b96fd81b6326aaa901682aa57720e519b6ea7415f237f9ff01298c6c23 WatchSource:0}: Error finding container fbb816b96fd81b6326aaa901682aa57720e519b6ea7415f237f9ff01298c6c23: Status 404 returned error can't find the container with id fbb816b96fd81b6326aaa901682aa57720e519b6ea7415f237f9ff01298c6c23
Sep 10 14:42:21.806201 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:21.806161    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7934-531ef0" event={"ID":"9b2349a1-ab3c-446e-b0cd-3f562c9277ce","Type":"ContainerStarted","Data":"bdb46026bef7b5933a5a83b6b800f192c90943d7f7d2c9fc60b42f779417834e"}
Sep 10 14:42:21.806201 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:21.806199    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7934-531ef0" event={"ID":"9b2349a1-ab3c-446e-b0cd-3f562c9277ce","Type":"ContainerStarted","Data":"fbb816b96fd81b6326aaa901682aa57720e519b6ea7415f237f9ff01298c6c23"}
Sep 10 14:42:21.822028 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:21.821958    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7934-531ef0" podStartSLOduration=1.821943084 podStartE2EDuration="1.821943084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:21.82004392 +0000 UTC m=+1360.011572118" watchObservedRunningTime="2026-09-10 14:42:21.821943084 +0000 UTC m=+1360.013471282"
Sep 10 14:42:22.387706 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:22.387666    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7934-531ef0"]
Sep 10 14:42:22.391758 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:22.391727    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7934-531ef0"]
Sep 10 14:42:22.510621 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:22.510536    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7934-6cc351"]
Sep 10 14:42:22.511071 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:22.511051    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b2349a1-ab3c-446e-b0cd-3f562c9277ce" containerName="test-curl-7934-531ef0"
Sep 10 14:42:22.511174 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:22.511074    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b2349a1-ab3c-446e-b0cd-3f562c9277ce" containerName="test-curl-7934-531ef0"
Sep 10 14:42:22.511174 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:22.511167    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b2349a1-ab3c-446e-b0cd-3f562c9277ce" containerName="test-curl-7934-531ef0"
Sep 10 14:42:22.514255 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:22.514236    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7934-6cc351"
Sep 10 14:42:22.521109 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:22.521082    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7934-6cc351"]
Sep 10 14:42:22.569912 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:22.569874    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rkpw\" (UniqueName: \"kubernetes.io/projected/05714de3-697f-45ac-a2da-d79c02a68acb-kube-api-access-8rkpw\") pod \"test-curl-7934-6cc351\" (UID: \"05714de3-697f-45ac-a2da-d79c02a68acb\") " pod="opendatahub/test-curl-7934-6cc351"
Sep 10 14:42:22.671527 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:22.671480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rkpw\" (UniqueName: \"kubernetes.io/projected/05714de3-697f-45ac-a2da-d79c02a68acb-kube-api-access-8rkpw\") pod \"test-curl-7934-6cc351\" (UID: \"05714de3-697f-45ac-a2da-d79c02a68acb\") " pod="opendatahub/test-curl-7934-6cc351"
Sep 10 14:42:22.679464 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:22.679434    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rkpw\" (UniqueName: \"kubernetes.io/projected/05714de3-697f-45ac-a2da-d79c02a68acb-kube-api-access-8rkpw\") pod \"test-curl-7934-6cc351\" (UID: \"05714de3-697f-45ac-a2da-d79c02a68acb\") " pod="opendatahub/test-curl-7934-6cc351"
Sep 10 14:42:22.812445 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:42:22.812359    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7934-531ef0" container="test-curl-7934-531ef0"
Sep 10 14:42:22.826120 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:22.826092    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7934-6cc351"
Sep 10 14:42:23.095181 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:23.095151    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7934-6cc351"]
Sep 10 14:42:23.096661 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:42:23.096635    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05714de3_697f_45ac_a2da_d79c02a68acb.slice/crio-5c080c03706394167ab67c5450e5f60af426db48c2081fc8e692864d53a28844 WatchSource:0}: Error finding container 5c080c03706394167ab67c5450e5f60af426db48c2081fc8e692864d53a28844: Status 404 returned error can't find the container with id 5c080c03706394167ab67c5450e5f60af426db48c2081fc8e692864d53a28844
Sep 10 14:42:23.820579 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:23.820541    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7934-6cc351" event={"ID":"05714de3-697f-45ac-a2da-d79c02a68acb","Type":"ContainerStarted","Data":"1bb275604f45ecfc828ed6df5bd1fc9ae6f8589cbf9005b6b76a5406d03ba48a"}
Sep 10 14:42:23.820579 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:23.820589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7934-6cc351" event={"ID":"05714de3-697f-45ac-a2da-d79c02a68acb","Type":"ContainerStarted","Data":"5c080c03706394167ab67c5450e5f60af426db48c2081fc8e692864d53a28844"}
Sep 10 14:42:23.821127 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:23.820628    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7934-531ef0" podUID="9b2349a1-ab3c-446e-b0cd-3f562c9277ce" containerName="test-curl-7934-531ef0" containerID="cri-o://bdb46026bef7b5933a5a83b6b800f192c90943d7f7d2c9fc60b42f779417834e" gracePeriod=2
Sep 10 14:42:23.834342 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:23.834294    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7934-6cc351" podStartSLOduration=1.83427886 podStartE2EDuration="1.83427886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:23.833084137 +0000 UTC m=+1362.024612348" watchObservedRunningTime="2026-09-10 14:42:23.83427886 +0000 UTC m=+1362.025807057"
Sep 10 14:42:23.834933 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:42:23.834905    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7934-531ef0\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="9b2349a1-ab3c-446e-b0cd-3f562c9277ce" pod="opendatahub/test-curl-7934-531ef0"
Sep 10 14:42:23.836815 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:42:23.836792    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7934-531ef0\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="9b2349a1-ab3c-446e-b0cd-3f562c9277ce" pod="opendatahub/test-curl-7934-531ef0"
Sep 10 14:42:24.395545 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:24.395513    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7934-6cc351"]
Sep 10 14:42:24.399553 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:24.399510    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7934-6cc351"]
Sep 10 14:42:24.826160 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:42:24.826128    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7934-6cc351" container="test-curl-7934-6cc351"
Sep 10 14:42:25.835653 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:25.835619    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7934-6cc351" podUID="05714de3-697f-45ac-a2da-d79c02a68acb" containerName="test-curl-7934-6cc351" containerID="cri-o://1bb275604f45ecfc828ed6df5bd1fc9ae6f8589cbf9005b6b76a5406d03ba48a" gracePeriod=2
Sep 10 14:42:25.838062 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:42:25.837929    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7934-6cc351\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="05714de3-697f-45ac-a2da-d79c02a68acb" pod="opendatahub/test-curl-7934-6cc351"
Sep 10 14:42:25.988135 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:25.988109    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7934-531ef0"
Sep 10 14:42:25.990937 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:42:25.990907    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7934-6cc351\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="05714de3-697f-45ac-a2da-d79c02a68acb" pod="opendatahub/test-curl-7934-6cc351"
Sep 10 14:42:26.108574 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:26.108488    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcgcf\" (UniqueName: \"kubernetes.io/projected/9b2349a1-ab3c-446e-b0cd-3f562c9277ce-kube-api-access-fcgcf\") pod \"9b2349a1-ab3c-446e-b0cd-3f562c9277ce\" (UID: \"9b2349a1-ab3c-446e-b0cd-3f562c9277ce\") "
Sep 10 14:42:26.110500 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:26.110472    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2349a1-ab3c-446e-b0cd-3f562c9277ce-kube-api-access-fcgcf" (OuterVolumeSpecName: "kube-api-access-fcgcf") pod "9b2349a1-ab3c-446e-b0cd-3f562c9277ce" (UID: "9b2349a1-ab3c-446e-b0cd-3f562c9277ce"). InnerVolumeSpecName "kube-api-access-fcgcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:42:26.210357 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:26.210307    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcgcf\" (UniqueName: \"kubernetes.io/projected/9b2349a1-ab3c-446e-b0cd-3f562c9277ce-kube-api-access-fcgcf\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:26.311273 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:26.311239    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b2349a1-ab3c-446e-b0cd-3f562c9277ce" path="/var/lib/kubelet/pods/9b2349a1-ab3c-446e-b0cd-3f562c9277ce/volumes"
Sep 10 14:42:26.372114 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:26.372031    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"]
Sep 10 14:42:26.842037 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:26.842002    2580 generic.go:356] "Generic (PLEG): container finished" podID="9b2349a1-ab3c-446e-b0cd-3f562c9277ce" containerID="bdb46026bef7b5933a5a83b6b800f192c90943d7f7d2c9fc60b42f779417834e" exitCode=137
Sep 10 14:42:26.842037 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:26.842027    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7934-531ef0"
Sep 10 14:42:26.842562 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:26.842079    2580 scope.go:117] "RemoveContainer" containerID="bdb46026bef7b5933a5a83b6b800f192c90943d7f7d2c9fc60b42f779417834e"
Sep 10 14:42:26.851413 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:26.851395    2580 scope.go:117] "RemoveContainer" containerID="bdb46026bef7b5933a5a83b6b800f192c90943d7f7d2c9fc60b42f779417834e"
Sep 10 14:42:26.851707 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:42:26.851686    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb46026bef7b5933a5a83b6b800f192c90943d7f7d2c9fc60b42f779417834e\": container with ID starting with bdb46026bef7b5933a5a83b6b800f192c90943d7f7d2c9fc60b42f779417834e not found: ID does not exist" containerID="bdb46026bef7b5933a5a83b6b800f192c90943d7f7d2c9fc60b42f779417834e"
Sep 10 14:42:26.851760 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:26.851716    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb46026bef7b5933a5a83b6b800f192c90943d7f7d2c9fc60b42f779417834e"} err="failed to get container status \"bdb46026bef7b5933a5a83b6b800f192c90943d7f7d2c9fc60b42f779417834e\": rpc error: code = NotFound desc = could not find container \"bdb46026bef7b5933a5a83b6b800f192c90943d7f7d2c9fc60b42f779417834e\": container with ID starting with bdb46026bef7b5933a5a83b6b800f192c90943d7f7d2c9fc60b42f779417834e not found: ID does not exist"
Sep 10 14:42:27.418718 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:27.418684    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"]
Sep 10 14:42:28.010360 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:28.010332    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7934-6cc351"
Sep 10 14:42:28.128621 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:28.128535    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rkpw\" (UniqueName: \"kubernetes.io/projected/05714de3-697f-45ac-a2da-d79c02a68acb-kube-api-access-8rkpw\") pod \"05714de3-697f-45ac-a2da-d79c02a68acb\" (UID: \"05714de3-697f-45ac-a2da-d79c02a68acb\") "
Sep 10 14:42:28.130571 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:28.130539    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05714de3-697f-45ac-a2da-d79c02a68acb-kube-api-access-8rkpw" (OuterVolumeSpecName: "kube-api-access-8rkpw") pod "05714de3-697f-45ac-a2da-d79c02a68acb" (UID: "05714de3-697f-45ac-a2da-d79c02a68acb"). InnerVolumeSpecName "kube-api-access-8rkpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:42:28.229414 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:28.229380    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rkpw\" (UniqueName: \"kubernetes.io/projected/05714de3-697f-45ac-a2da-d79c02a68acb-kube-api-access-8rkpw\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:28.311754 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:28.311721    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05714de3-697f-45ac-a2da-d79c02a68acb" path="/var/lib/kubelet/pods/05714de3-697f-45ac-a2da-d79c02a68acb/volumes"
Sep 10 14:42:28.855980 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:28.855930    2580 generic.go:356] "Generic (PLEG): container finished" podID="05714de3-697f-45ac-a2da-d79c02a68acb" containerID="1bb275604f45ecfc828ed6df5bd1fc9ae6f8589cbf9005b6b76a5406d03ba48a" exitCode=137
Sep 10 14:42:28.856164 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:28.855994    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7934-6cc351"
Sep 10 14:42:28.856164 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:28.856026    2580 scope.go:117] "RemoveContainer" containerID="1bb275604f45ecfc828ed6df5bd1fc9ae6f8589cbf9005b6b76a5406d03ba48a"
Sep 10 14:42:28.865264 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:28.865245    2580 scope.go:117] "RemoveContainer" containerID="1bb275604f45ecfc828ed6df5bd1fc9ae6f8589cbf9005b6b76a5406d03ba48a"
Sep 10 14:42:28.865529 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:42:28.865510    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb275604f45ecfc828ed6df5bd1fc9ae6f8589cbf9005b6b76a5406d03ba48a\": container with ID starting with 1bb275604f45ecfc828ed6df5bd1fc9ae6f8589cbf9005b6b76a5406d03ba48a not found: ID does not exist" containerID="1bb275604f45ecfc828ed6df5bd1fc9ae6f8589cbf9005b6b76a5406d03ba48a"
Sep 10 14:42:28.865574 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:28.865537    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb275604f45ecfc828ed6df5bd1fc9ae6f8589cbf9005b6b76a5406d03ba48a"} err="failed to get container status \"1bb275604f45ecfc828ed6df5bd1fc9ae6f8589cbf9005b6b76a5406d03ba48a\": rpc error: code = NotFound desc = could not find container \"1bb275604f45ecfc828ed6df5bd1fc9ae6f8589cbf9005b6b76a5406d03ba48a\": container with ID starting with 1bb275604f45ecfc828ed6df5bd1fc9ae6f8589cbf9005b6b76a5406d03ba48a not found: ID does not exist"
Sep 10 14:42:38.116403 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.116368    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"]
Sep 10 14:42:38.116805 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.116791    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05714de3-697f-45ac-a2da-d79c02a68acb" containerName="test-curl-7934-6cc351"
Sep 10 14:42:38.116851 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.116806    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="05714de3-697f-45ac-a2da-d79c02a68acb" containerName="test-curl-7934-6cc351"
Sep 10 14:42:38.116892 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.116876    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="05714de3-697f-45ac-a2da-d79c02a68acb" containerName="test-curl-7934-6cc351"
Sep 10 14:42:38.121591 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.121574    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.124136 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.124114    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7bf16f\"/\"rate-test-model-7bf16f-kserve-self-signed-certs\""
Sep 10 14:42:38.128496 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.128474    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"]
Sep 10 14:42:38.228572 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.228531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt97p\" (UniqueName: \"kubernetes.io/projected/1e14836f-1145-41ee-867a-54b7afe8811b-kube-api-access-kt97p\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.228723 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.228635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-home\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.228723 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.228689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e14836f-1145-41ee-867a-54b7afe8811b-tls-certs\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.228723 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.228716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-tmp-dir\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.228884 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.228863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-dshm\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.228928 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.228902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-model-cache\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.329526 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.329496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-home\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.329698 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.329536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e14836f-1145-41ee-867a-54b7afe8811b-tls-certs\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.329698 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.329554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-tmp-dir\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.329698 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.329606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-dshm\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.329698 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.329626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-model-cache\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.329931 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.329777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt97p\" (UniqueName: \"kubernetes.io/projected/1e14836f-1145-41ee-867a-54b7afe8811b-kube-api-access-kt97p\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.329931 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.329921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-home\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.330078 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.330024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-tmp-dir\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.330078 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.330054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-model-cache\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.331914 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.331889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-dshm\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.332141 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.332125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e14836f-1145-41ee-867a-54b7afe8811b-tls-certs\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.338243 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.338215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt97p\" (UniqueName: \"kubernetes.io/projected/1e14836f-1145-41ee-867a-54b7afe8811b-kube-api-access-kt97p\") pod \"rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") " pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.431561 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.431528    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:38.705804 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.705774    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"]
Sep 10 14:42:38.707834 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:42:38.707805    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e14836f_1145_41ee_867a_54b7afe8811b.slice/crio-1ec9491724375ea53e928dc427953a8aa00ec69d69c186c14a8fbd6e010b5d29 WatchSource:0}: Error finding container 1ec9491724375ea53e928dc427953a8aa00ec69d69c186c14a8fbd6e010b5d29: Status 404 returned error can't find the container with id 1ec9491724375ea53e928dc427953a8aa00ec69d69c186c14a8fbd6e010b5d29
Sep 10 14:42:38.913780 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.913739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5" event={"ID":"1e14836f-1145-41ee-867a-54b7afe8811b","Type":"ContainerStarted","Data":"3bd608381b9c18963f7d47062b92b358e99c9075eac54daa3157912e2eb3dfbf"}
Sep 10 14:42:38.913962 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.913789    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5" event={"ID":"1e14836f-1145-41ee-867a-54b7afe8811b","Type":"ContainerStarted","Data":"1ec9491724375ea53e928dc427953a8aa00ec69d69c186c14a8fbd6e010b5d29"}
Sep 10 14:42:38.932968 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:38.932908    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5" podStartSLOduration=0.93289114 podStartE2EDuration="932.89114ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:38.930126818 +0000 UTC m=+1377.121655016" watchObservedRunningTime="2026-09-10 14:42:38.93289114 +0000 UTC m=+1377.124419337"
Sep 10 14:42:41.392186 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.392127    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf" podUID="68a66be9-b80f-4210-b86e-577d581b0209" containerName="main" containerID="cri-o://dd483b65ece6de4360f97a968f26f0221ca1540507fa8fd77ef831c6c9897567" gracePeriod=45
Sep 10 14:42:41.645271 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.645213    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:42:41.765555 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.765518    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-model-cache\") pod \"68a66be9-b80f-4210-b86e-577d581b0209\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") "
Sep 10 14:42:41.765738 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.765573    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-dshm\") pod \"68a66be9-b80f-4210-b86e-577d581b0209\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") "
Sep 10 14:42:41.765738 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.765592    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-tmp-dir\") pod \"68a66be9-b80f-4210-b86e-577d581b0209\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") "
Sep 10 14:42:41.765738 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.765618    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc8nh\" (UniqueName: \"kubernetes.io/projected/68a66be9-b80f-4210-b86e-577d581b0209-kube-api-access-rc8nh\") pod \"68a66be9-b80f-4210-b86e-577d581b0209\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") "
Sep 10 14:42:41.765738 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.765647    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68a66be9-b80f-4210-b86e-577d581b0209-tls-certs\") pod \"68a66be9-b80f-4210-b86e-577d581b0209\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") "
Sep 10 14:42:41.765738 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.765726    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-home\") pod \"68a66be9-b80f-4210-b86e-577d581b0209\" (UID: \"68a66be9-b80f-4210-b86e-577d581b0209\") "
Sep 10 14:42:41.766030 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.765862    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-model-cache" (OuterVolumeSpecName: "model-cache") pod "68a66be9-b80f-4210-b86e-577d581b0209" (UID: "68a66be9-b80f-4210-b86e-577d581b0209"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:41.766030 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.765919    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68a66be9-b80f-4210-b86e-577d581b0209" (UID: "68a66be9-b80f-4210-b86e-577d581b0209"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:41.766131 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.766083    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-home" (OuterVolumeSpecName: "home") pod "68a66be9-b80f-4210-b86e-577d581b0209" (UID: "68a66be9-b80f-4210-b86e-577d581b0209"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:41.766165 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.766134    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:41.766165 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.766154    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:41.767681 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.767660    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-dshm" (OuterVolumeSpecName: "dshm") pod "68a66be9-b80f-4210-b86e-577d581b0209" (UID: "68a66be9-b80f-4210-b86e-577d581b0209"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:41.768233 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.768208    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68a66be9-b80f-4210-b86e-577d581b0209-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68a66be9-b80f-4210-b86e-577d581b0209" (UID: "68a66be9-b80f-4210-b86e-577d581b0209"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:42:41.768335 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.768237    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a66be9-b80f-4210-b86e-577d581b0209-kube-api-access-rc8nh" (OuterVolumeSpecName: "kube-api-access-rc8nh") pod "68a66be9-b80f-4210-b86e-577d581b0209" (UID: "68a66be9-b80f-4210-b86e-577d581b0209"). InnerVolumeSpecName "kube-api-access-rc8nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:42:41.867201 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.867165    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:41.867201 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.867192    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68a66be9-b80f-4210-b86e-577d581b0209-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:41.867201 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.867202    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc8nh\" (UniqueName: \"kubernetes.io/projected/68a66be9-b80f-4210-b86e-577d581b0209-kube-api-access-rc8nh\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:41.867446 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.867212    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68a66be9-b80f-4210-b86e-577d581b0209-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:41.932290 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.932206    2580 generic.go:356] "Generic (PLEG): container finished" podID="68a66be9-b80f-4210-b86e-577d581b0209" containerID="dd483b65ece6de4360f97a968f26f0221ca1540507fa8fd77ef831c6c9897567" exitCode=0
Sep 10 14:42:41.932290 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.932245    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"
Sep 10 14:42:41.932473 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.932294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf" event={"ID":"68a66be9-b80f-4210-b86e-577d581b0209","Type":"ContainerDied","Data":"dd483b65ece6de4360f97a968f26f0221ca1540507fa8fd77ef831c6c9897567"}
Sep 10 14:42:41.932473 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.932333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf" event={"ID":"68a66be9-b80f-4210-b86e-577d581b0209","Type":"ContainerDied","Data":"09bcb0d66dff5c2d3afc46a4bdf9614904654147eccc0b231525a93342917107"}
Sep 10 14:42:41.932473 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.932355    2580 scope.go:117] "RemoveContainer" containerID="dd483b65ece6de4360f97a968f26f0221ca1540507fa8fd77ef831c6c9897567"
Sep 10 14:42:41.941989 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.941954    2580 scope.go:117] "RemoveContainer" containerID="dd483b65ece6de4360f97a968f26f0221ca1540507fa8fd77ef831c6c9897567"
Sep 10 14:42:41.942273 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:42:41.942243    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd483b65ece6de4360f97a968f26f0221ca1540507fa8fd77ef831c6c9897567\": container with ID starting with dd483b65ece6de4360f97a968f26f0221ca1540507fa8fd77ef831c6c9897567 not found: ID does not exist" containerID="dd483b65ece6de4360f97a968f26f0221ca1540507fa8fd77ef831c6c9897567"
Sep 10 14:42:41.942318 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.942280    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd483b65ece6de4360f97a968f26f0221ca1540507fa8fd77ef831c6c9897567"} err="failed to get container status \"dd483b65ece6de4360f97a968f26f0221ca1540507fa8fd77ef831c6c9897567\": rpc error: code = NotFound desc = could not find container \"dd483b65ece6de4360f97a968f26f0221ca1540507fa8fd77ef831c6c9897567\": container with ID starting with dd483b65ece6de4360f97a968f26f0221ca1540507fa8fd77ef831c6c9897567 not found: ID does not exist"
Sep 10 14:42:41.953379 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.953351    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"]
Sep 10 14:42:41.959024 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:41.959004    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/sub-test-model-7bf16f-kserve-54f9d88995-s52rf"]
Sep 10 14:42:42.311572 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.311485    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68a66be9-b80f-4210-b86e-577d581b0209" path="/var/lib/kubelet/pods/68a66be9-b80f-4210-b86e-577d581b0209/volumes"
Sep 10 14:42:42.439481 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.439447    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q" podUID="4d4dccd6-c979-4621-adca-f69a990ffa5e" containerName="main" containerID="cri-o://9d196bba007beba6699c12b021d0c0d4ff0b71796f65ac5f118953c76bed2a6c" gracePeriod=45
Sep 10 14:42:42.700913 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.700887    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:42:42.877040 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.876947    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-model-cache\") pod \"4d4dccd6-c979-4621-adca-f69a990ffa5e\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") "
Sep 10 14:42:42.877040 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.877032    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5bnb\" (UniqueName: \"kubernetes.io/projected/4d4dccd6-c979-4621-adca-f69a990ffa5e-kube-api-access-q5bnb\") pod \"4d4dccd6-c979-4621-adca-f69a990ffa5e\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") "
Sep 10 14:42:42.877324 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.877096    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4dccd6-c979-4621-adca-f69a990ffa5e-tls-certs\") pod \"4d4dccd6-c979-4621-adca-f69a990ffa5e\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") "
Sep 10 14:42:42.877324 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.877135    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-dshm\") pod \"4d4dccd6-c979-4621-adca-f69a990ffa5e\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") "
Sep 10 14:42:42.877324 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.877202    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-home\") pod \"4d4dccd6-c979-4621-adca-f69a990ffa5e\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") "
Sep 10 14:42:42.877324 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.877237    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-tmp-dir\") pod \"4d4dccd6-c979-4621-adca-f69a990ffa5e\" (UID: \"4d4dccd6-c979-4621-adca-f69a990ffa5e\") "
Sep 10 14:42:42.877324 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.877254    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d4dccd6-c979-4621-adca-f69a990ffa5e" (UID: "4d4dccd6-c979-4621-adca-f69a990ffa5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:42.877592 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.877505    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-home" (OuterVolumeSpecName: "home") pod "4d4dccd6-c979-4621-adca-f69a990ffa5e" (UID: "4d4dccd6-c979-4621-adca-f69a990ffa5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:42.877592 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.877519    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d4dccd6-c979-4621-adca-f69a990ffa5e" (UID: "4d4dccd6-c979-4621-adca-f69a990ffa5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:42.877701 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.877619    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:42.877701 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.877634    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:42.877701 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.877642    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:42.879393 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.879364    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-dshm" (OuterVolumeSpecName: "dshm") pod "4d4dccd6-c979-4621-adca-f69a990ffa5e" (UID: "4d4dccd6-c979-4621-adca-f69a990ffa5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:42:42.879534 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.879404    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4dccd6-c979-4621-adca-f69a990ffa5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d4dccd6-c979-4621-adca-f69a990ffa5e" (UID: "4d4dccd6-c979-4621-adca-f69a990ffa5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:42:42.879534 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.879422    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4dccd6-c979-4621-adca-f69a990ffa5e-kube-api-access-q5bnb" (OuterVolumeSpecName: "kube-api-access-q5bnb") pod "4d4dccd6-c979-4621-adca-f69a990ffa5e" (UID: "4d4dccd6-c979-4621-adca-f69a990ffa5e"). InnerVolumeSpecName "kube-api-access-q5bnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:42:42.939550 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.939519    2580 generic.go:356] "Generic (PLEG): container finished" podID="4d4dccd6-c979-4621-adca-f69a990ffa5e" containerID="9d196bba007beba6699c12b021d0c0d4ff0b71796f65ac5f118953c76bed2a6c" exitCode=0
Sep 10 14:42:42.939720 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.939562    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"
Sep 10 14:42:42.939720 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.939566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q" event={"ID":"4d4dccd6-c979-4621-adca-f69a990ffa5e","Type":"ContainerDied","Data":"9d196bba007beba6699c12b021d0c0d4ff0b71796f65ac5f118953c76bed2a6c"}
Sep 10 14:42:42.939720 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.939689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q" event={"ID":"4d4dccd6-c979-4621-adca-f69a990ffa5e","Type":"ContainerDied","Data":"881a6897d698d470c2160ec1e29d95a6047998c007fc3791b9396d27fde73915"}
Sep 10 14:42:42.939720 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.939716    2580 scope.go:117] "RemoveContainer" containerID="9d196bba007beba6699c12b021d0c0d4ff0b71796f65ac5f118953c76bed2a6c"
Sep 10 14:42:42.948916 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.948897    2580 scope.go:117] "RemoveContainer" containerID="9d196bba007beba6699c12b021d0c0d4ff0b71796f65ac5f118953c76bed2a6c"
Sep 10 14:42:42.949195 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:42:42.949178    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d196bba007beba6699c12b021d0c0d4ff0b71796f65ac5f118953c76bed2a6c\": container with ID starting with 9d196bba007beba6699c12b021d0c0d4ff0b71796f65ac5f118953c76bed2a6c not found: ID does not exist" containerID="9d196bba007beba6699c12b021d0c0d4ff0b71796f65ac5f118953c76bed2a6c"
Sep 10 14:42:42.949251 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.949202    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d196bba007beba6699c12b021d0c0d4ff0b71796f65ac5f118953c76bed2a6c"} err="failed to get container status \"9d196bba007beba6699c12b021d0c0d4ff0b71796f65ac5f118953c76bed2a6c\": rpc error: code = NotFound desc = could not find container \"9d196bba007beba6699c12b021d0c0d4ff0b71796f65ac5f118953c76bed2a6c\": container with ID starting with 9d196bba007beba6699c12b021d0c0d4ff0b71796f65ac5f118953c76bed2a6c not found: ID does not exist"
Sep 10 14:42:42.961273 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.961244    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"]
Sep 10 14:42:42.967431 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.967400    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/sub-test-model-59dc59-kserve-865f8f64b9-nkr4q"]
Sep 10 14:42:42.978334 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.978309    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5bnb\" (UniqueName: \"kubernetes.io/projected/4d4dccd6-c979-4621-adca-f69a990ffa5e-kube-api-access-q5bnb\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:42.978334 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.978333    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4dccd6-c979-4621-adca-f69a990ffa5e-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:42.978452 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:42.978342    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4dccd6-c979-4621-adca-f69a990ffa5e-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:44.310689 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:44.310659    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d4dccd6-c979-4621-adca-f69a990ffa5e" path="/var/lib/kubelet/pods/4d4dccd6-c979-4621-adca-f69a990ffa5e/volumes"
Sep 10 14:42:47.140946 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.140898    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"]
Sep 10 14:42:47.141428 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.141182    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh" podUID="7383a8f2-1883-4022-9b81-aec86c647be0" containerName="maas-api" containerID="cri-o://ae2990970e2a00980ff5df50ed4a3e53855fd463d087742ff5765ed24d201348" gracePeriod=30
Sep 10 14:42:47.388755 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.388727    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:42:47.425957 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.425875    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7383a8f2-1883-4022-9b81-aec86c647be0-openshift-service-ca\") pod \"7383a8f2-1883-4022-9b81-aec86c647be0\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") "
Sep 10 14:42:47.426140 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.426052    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jcr8\" (UniqueName: \"kubernetes.io/projected/7383a8f2-1883-4022-9b81-aec86c647be0-kube-api-access-9jcr8\") pod \"7383a8f2-1883-4022-9b81-aec86c647be0\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") "
Sep 10 14:42:47.426140 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.426092    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7383a8f2-1883-4022-9b81-aec86c647be0-maas-api-tls\") pod \"7383a8f2-1883-4022-9b81-aec86c647be0\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") "
Sep 10 14:42:47.426140 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.426122    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7383a8f2-1883-4022-9b81-aec86c647be0-metrics-certs\") pod \"7383a8f2-1883-4022-9b81-aec86c647be0\" (UID: \"7383a8f2-1883-4022-9b81-aec86c647be0\") "
Sep 10 14:42:47.426317 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.426283    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7383a8f2-1883-4022-9b81-aec86c647be0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7383a8f2-1883-4022-9b81-aec86c647be0" (UID: "7383a8f2-1883-4022-9b81-aec86c647be0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:42:47.426470 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.426443    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7383a8f2-1883-4022-9b81-aec86c647be0-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:47.428362 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.428333    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7383a8f2-1883-4022-9b81-aec86c647be0-kube-api-access-9jcr8" (OuterVolumeSpecName: "kube-api-access-9jcr8") pod "7383a8f2-1883-4022-9b81-aec86c647be0" (UID: "7383a8f2-1883-4022-9b81-aec86c647be0"). InnerVolumeSpecName "kube-api-access-9jcr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:42:47.428710 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.428690    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7383a8f2-1883-4022-9b81-aec86c647be0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7383a8f2-1883-4022-9b81-aec86c647be0" (UID: "7383a8f2-1883-4022-9b81-aec86c647be0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:42:47.428710 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.428701    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7383a8f2-1883-4022-9b81-aec86c647be0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7383a8f2-1883-4022-9b81-aec86c647be0" (UID: "7383a8f2-1883-4022-9b81-aec86c647be0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:42:47.527447 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.527406    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jcr8\" (UniqueName: \"kubernetes.io/projected/7383a8f2-1883-4022-9b81-aec86c647be0-kube-api-access-9jcr8\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:47.527447 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.527445    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7383a8f2-1883-4022-9b81-aec86c647be0-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:47.527447 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.527455    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7383a8f2-1883-4022-9b81-aec86c647be0-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:42:47.971286 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.971257    2580 generic.go:356] "Generic (PLEG): container finished" podID="7383a8f2-1883-4022-9b81-aec86c647be0" containerID="ae2990970e2a00980ff5df50ed4a3e53855fd463d087742ff5765ed24d201348" exitCode=0
Sep 10 14:42:47.971474 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.971298    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"
Sep 10 14:42:47.971474 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.971341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh" event={"ID":"7383a8f2-1883-4022-9b81-aec86c647be0","Type":"ContainerDied","Data":"ae2990970e2a00980ff5df50ed4a3e53855fd463d087742ff5765ed24d201348"}
Sep 10 14:42:47.971474 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.971395    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh" event={"ID":"7383a8f2-1883-4022-9b81-aec86c647be0","Type":"ContainerDied","Data":"7ce87add9a16dda8c462f586e5a33c877173046f841135e78698b4e4e2492e7b"}
Sep 10 14:42:47.971474 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.971422    2580 scope.go:117] "RemoveContainer" containerID="ae2990970e2a00980ff5df50ed4a3e53855fd463d087742ff5765ed24d201348"
Sep 10 14:42:47.981076 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.981059    2580 scope.go:117] "RemoveContainer" containerID="ae2990970e2a00980ff5df50ed4a3e53855fd463d087742ff5765ed24d201348"
Sep 10 14:42:47.981356 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:42:47.981337    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2990970e2a00980ff5df50ed4a3e53855fd463d087742ff5765ed24d201348\": container with ID starting with ae2990970e2a00980ff5df50ed4a3e53855fd463d087742ff5765ed24d201348 not found: ID does not exist" containerID="ae2990970e2a00980ff5df50ed4a3e53855fd463d087742ff5765ed24d201348"
Sep 10 14:42:47.981405 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.981365    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2990970e2a00980ff5df50ed4a3e53855fd463d087742ff5765ed24d201348"} err="failed to get container status \"ae2990970e2a00980ff5df50ed4a3e53855fd463d087742ff5765ed24d201348\": rpc error: code = NotFound desc = could not find container \"ae2990970e2a00980ff5df50ed4a3e53855fd463d087742ff5765ed24d201348\": container with ID starting with ae2990970e2a00980ff5df50ed4a3e53855fd463d087742ff5765ed24d201348 not found: ID does not exist"
Sep 10 14:42:47.993272 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.993247    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"]
Sep 10 14:42:47.997745 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:47.997724    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1d2af2-546d446df8-7nzjh"]
Sep 10 14:42:48.310740 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:48.310659    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7383a8f2-1883-4022-9b81-aec86c647be0" path="/var/lib/kubelet/pods/7383a8f2-1883-4022-9b81-aec86c647be0/volumes"
Sep 10 14:42:48.432193 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:48.432162    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:48.432363 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:48.432209    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:48.444780 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:48.444752    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:42:48.988888 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:42:48.988857    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:43:09.833950 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:09.833908    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"]
Sep 10 14:43:09.834580 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:09.834174    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn" podUID="141612d7-93f1-437b-b876-fdcba29cc166" containerName="maas-api" containerID="cri-o://e32025361338ef9a3f7d60b38d8dca5910a3eef237487b7dfa3132c04f488efc" gracePeriod=30
Sep 10 14:43:10.098495 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.098468    2580 generic.go:356] "Generic (PLEG): container finished" podID="141612d7-93f1-437b-b876-fdcba29cc166" containerID="e32025361338ef9a3f7d60b38d8dca5910a3eef237487b7dfa3132c04f488efc" exitCode=0
Sep 10 14:43:10.098634 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.098544    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn" event={"ID":"141612d7-93f1-437b-b876-fdcba29cc166","Type":"ContainerDied","Data":"e32025361338ef9a3f7d60b38d8dca5910a3eef237487b7dfa3132c04f488efc"}
Sep 10 14:43:10.098634 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.098597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn" event={"ID":"141612d7-93f1-437b-b876-fdcba29cc166","Type":"ContainerDied","Data":"500401ec99534b3c7ed1f789c67aa907f5471ad45e30c6d3c05bcf9fc3616c02"}
Sep 10 14:43:10.098634 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.098615    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="500401ec99534b3c7ed1f789c67aa907f5471ad45e30c6d3c05bcf9fc3616c02"
Sep 10 14:43:10.105240 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.105219    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:43:10.260173 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.260130    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/141612d7-93f1-437b-b876-fdcba29cc166-metrics-certs\") pod \"141612d7-93f1-437b-b876-fdcba29cc166\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") "
Sep 10 14:43:10.260375 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.260191    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzpsz\" (UniqueName: \"kubernetes.io/projected/141612d7-93f1-437b-b876-fdcba29cc166-kube-api-access-mzpsz\") pod \"141612d7-93f1-437b-b876-fdcba29cc166\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") "
Sep 10 14:43:10.260375 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.260216    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141612d7-93f1-437b-b876-fdcba29cc166-maas-api-tls\") pod \"141612d7-93f1-437b-b876-fdcba29cc166\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") "
Sep 10 14:43:10.260375 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.260249    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141612d7-93f1-437b-b876-fdcba29cc166-openshift-service-ca\") pod \"141612d7-93f1-437b-b876-fdcba29cc166\" (UID: \"141612d7-93f1-437b-b876-fdcba29cc166\") "
Sep 10 14:43:10.260698 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.260667    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141612d7-93f1-437b-b876-fdcba29cc166-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "141612d7-93f1-437b-b876-fdcba29cc166" (UID: "141612d7-93f1-437b-b876-fdcba29cc166"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:43:10.262403 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.262378    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141612d7-93f1-437b-b876-fdcba29cc166-kube-api-access-mzpsz" (OuterVolumeSpecName: "kube-api-access-mzpsz") pod "141612d7-93f1-437b-b876-fdcba29cc166" (UID: "141612d7-93f1-437b-b876-fdcba29cc166"). InnerVolumeSpecName "kube-api-access-mzpsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:43:10.262403 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.262385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141612d7-93f1-437b-b876-fdcba29cc166-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "141612d7-93f1-437b-b876-fdcba29cc166" (UID: "141612d7-93f1-437b-b876-fdcba29cc166"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:43:10.262540 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.262429    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141612d7-93f1-437b-b876-fdcba29cc166-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "141612d7-93f1-437b-b876-fdcba29cc166" (UID: "141612d7-93f1-437b-b876-fdcba29cc166"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:43:10.361460 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.361391    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/141612d7-93f1-437b-b876-fdcba29cc166-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:43:10.361460 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.361414    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzpsz\" (UniqueName: \"kubernetes.io/projected/141612d7-93f1-437b-b876-fdcba29cc166-kube-api-access-mzpsz\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:43:10.361460 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.361424    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141612d7-93f1-437b-b876-fdcba29cc166-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:43:10.361460 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:10.361434    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141612d7-93f1-437b-b876-fdcba29cc166-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:43:11.103571 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:11.103540    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"
Sep 10 14:43:11.123512 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:11.123480    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"]
Sep 10 14:43:11.132383 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:11.132359    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ba551-68f577bfcc-z84rn"]
Sep 10 14:43:12.311894 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:12.311862    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="141612d7-93f1-437b-b876-fdcba29cc166" path="/var/lib/kubelet/pods/141612d7-93f1-437b-b876-fdcba29cc166/volumes"
Sep 10 14:43:27.475747 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:27.475714    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"]
Sep 10 14:43:27.476316 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:27.475958    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k" podUID="824244fd-f32d-4c9f-b0e7-321b28eeb26b" containerName="maas-api" containerID="cri-o://4f75bbe7a505fca0f1c9242ff1fad5495f3367a9b42baf4e327f8da1a9898db4" gracePeriod=30
Sep 10 14:43:27.734699 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:27.734637    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:43:27.822647 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:27.822615    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cq9h\" (UniqueName: \"kubernetes.io/projected/824244fd-f32d-4c9f-b0e7-321b28eeb26b-kube-api-access-5cq9h\") pod \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") "
Sep 10 14:43:27.822820 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:27.822699    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824244fd-f32d-4c9f-b0e7-321b28eeb26b-openshift-service-ca\") pod \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") "
Sep 10 14:43:27.822820 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:27.822722    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/824244fd-f32d-4c9f-b0e7-321b28eeb26b-metrics-certs\") pod \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") "
Sep 10 14:43:27.822820 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:27.822783    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824244fd-f32d-4c9f-b0e7-321b28eeb26b-maas-api-tls\") pod \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\" (UID: \"824244fd-f32d-4c9f-b0e7-321b28eeb26b\") "
Sep 10 14:43:27.823187 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:27.823151    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824244fd-f32d-4c9f-b0e7-321b28eeb26b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "824244fd-f32d-4c9f-b0e7-321b28eeb26b" (UID: "824244fd-f32d-4c9f-b0e7-321b28eeb26b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:43:27.824874 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:27.824848    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824244fd-f32d-4c9f-b0e7-321b28eeb26b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "824244fd-f32d-4c9f-b0e7-321b28eeb26b" (UID: "824244fd-f32d-4c9f-b0e7-321b28eeb26b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:43:27.825283 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:27.825251    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824244fd-f32d-4c9f-b0e7-321b28eeb26b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "824244fd-f32d-4c9f-b0e7-321b28eeb26b" (UID: "824244fd-f32d-4c9f-b0e7-321b28eeb26b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:43:27.825283 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:27.825264    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824244fd-f32d-4c9f-b0e7-321b28eeb26b-kube-api-access-5cq9h" (OuterVolumeSpecName: "kube-api-access-5cq9h") pod "824244fd-f32d-4c9f-b0e7-321b28eeb26b" (UID: "824244fd-f32d-4c9f-b0e7-321b28eeb26b"). InnerVolumeSpecName "kube-api-access-5cq9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:43:27.924189 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:27.924149    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824244fd-f32d-4c9f-b0e7-321b28eeb26b-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:43:27.924189 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:27.924180    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/824244fd-f32d-4c9f-b0e7-321b28eeb26b-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:43:27.924189 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:27.924190    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824244fd-f32d-4c9f-b0e7-321b28eeb26b-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:43:27.924189 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:27.924198    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cq9h\" (UniqueName: \"kubernetes.io/projected/824244fd-f32d-4c9f-b0e7-321b28eeb26b-kube-api-access-5cq9h\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:43:28.199827 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:28.199792    2580 generic.go:356] "Generic (PLEG): container finished" podID="824244fd-f32d-4c9f-b0e7-321b28eeb26b" containerID="4f75bbe7a505fca0f1c9242ff1fad5495f3367a9b42baf4e327f8da1a9898db4" exitCode=0
Sep 10 14:43:28.200028 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:28.199839    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"
Sep 10 14:43:28.200028 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:28.199839    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k" event={"ID":"824244fd-f32d-4c9f-b0e7-321b28eeb26b","Type":"ContainerDied","Data":"4f75bbe7a505fca0f1c9242ff1fad5495f3367a9b42baf4e327f8da1a9898db4"}
Sep 10 14:43:28.200028 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:28.200021    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k" event={"ID":"824244fd-f32d-4c9f-b0e7-321b28eeb26b","Type":"ContainerDied","Data":"43ed80da171fa6257bc24b36c114723a71b6abc8baa95079580e30fe6f0a68ff"}
Sep 10 14:43:28.200201 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:28.200053    2580 scope.go:117] "RemoveContainer" containerID="4f75bbe7a505fca0f1c9242ff1fad5495f3367a9b42baf4e327f8da1a9898db4"
Sep 10 14:43:28.209886 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:28.209870    2580 scope.go:117] "RemoveContainer" containerID="4f75bbe7a505fca0f1c9242ff1fad5495f3367a9b42baf4e327f8da1a9898db4"
Sep 10 14:43:28.210209 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:43:28.210189    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f75bbe7a505fca0f1c9242ff1fad5495f3367a9b42baf4e327f8da1a9898db4\": container with ID starting with 4f75bbe7a505fca0f1c9242ff1fad5495f3367a9b42baf4e327f8da1a9898db4 not found: ID does not exist" containerID="4f75bbe7a505fca0f1c9242ff1fad5495f3367a9b42baf4e327f8da1a9898db4"
Sep 10 14:43:28.210255 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:28.210218    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f75bbe7a505fca0f1c9242ff1fad5495f3367a9b42baf4e327f8da1a9898db4"} err="failed to get container status \"4f75bbe7a505fca0f1c9242ff1fad5495f3367a9b42baf4e327f8da1a9898db4\": rpc error: code = NotFound desc = could not find container \"4f75bbe7a505fca0f1c9242ff1fad5495f3367a9b42baf4e327f8da1a9898db4\": container with ID starting with 4f75bbe7a505fca0f1c9242ff1fad5495f3367a9b42baf4e327f8da1a9898db4 not found: ID does not exist"
Sep 10 14:43:28.221857 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:28.221832    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"]
Sep 10 14:43:28.227373 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:28.227350    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ecc1f-68967d6b45-dlc5k"]
Sep 10 14:43:28.310394 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:28.310357    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="824244fd-f32d-4c9f-b0e7-321b28eeb26b" path="/var/lib/kubelet/pods/824244fd-f32d-4c9f-b0e7-321b28eeb26b/volumes"
Sep 10 14:43:37.701878 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.701800    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"]
Sep 10 14:43:37.702456 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.702433    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d4dccd6-c979-4621-adca-f69a990ffa5e" containerName="main"
Sep 10 14:43:37.702562 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.702460    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d4dccd6-c979-4621-adca-f69a990ffa5e" containerName="main"
Sep 10 14:43:37.702562 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.702486    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68a66be9-b80f-4210-b86e-577d581b0209" containerName="main"
Sep 10 14:43:37.702562 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.702495    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="68a66be9-b80f-4210-b86e-577d581b0209" containerName="main"
Sep 10 14:43:37.702562 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.702518    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="824244fd-f32d-4c9f-b0e7-321b28eeb26b" containerName="maas-api"
Sep 10 14:43:37.702562 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.702529    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="824244fd-f32d-4c9f-b0e7-321b28eeb26b" containerName="maas-api"
Sep 10 14:43:37.702562 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.702541    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7383a8f2-1883-4022-9b81-aec86c647be0" containerName="maas-api"
Sep 10 14:43:37.702562 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.702549    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7383a8f2-1883-4022-9b81-aec86c647be0" containerName="maas-api"
Sep 10 14:43:37.702562 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.702558    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="141612d7-93f1-437b-b876-fdcba29cc166" containerName="maas-api"
Sep 10 14:43:37.702562 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.702566    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="141612d7-93f1-437b-b876-fdcba29cc166" containerName="maas-api"
Sep 10 14:43:37.703062 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.702648    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="824244fd-f32d-4c9f-b0e7-321b28eeb26b" containerName="maas-api"
Sep 10 14:43:37.703062 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.702657    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7383a8f2-1883-4022-9b81-aec86c647be0" containerName="maas-api"
Sep 10 14:43:37.703062 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.702662    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d4dccd6-c979-4621-adca-f69a990ffa5e" containerName="main"
Sep 10 14:43:37.703062 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.702669    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="141612d7-93f1-437b-b876-fdcba29cc166" containerName="maas-api"
Sep 10 14:43:37.703062 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.702677    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="68a66be9-b80f-4210-b86e-577d581b0209" containerName="main"
Sep 10 14:43:37.707272 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.707250    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.709808 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.709785    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-59dc59\"/\"kube-root-ca.crt\""
Sep 10 14:43:37.710796 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.710771    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-59dc59\"/\"rate-test-model-59dc59-kserve-self-signed-certs\""
Sep 10 14:43:37.710919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.710800    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-59dc59\"/\"openshift-service-ca.crt\""
Sep 10 14:43:37.711031 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.710925    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-59dc59\"/\"default-dockercfg-kcnjl\""
Sep 10 14:43:37.716867 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.716839    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"]
Sep 10 14:43:37.816833 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.816803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-tmp-dir\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.816833 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.816835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c4de614-4360-4e42-bbe2-2d5f60c123b3-tls-certs\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.817129 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.816862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-home\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.817129 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.816956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-dshm\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.817129 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.817092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-model-cache\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.817265 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.817147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s648q\" (UniqueName: \"kubernetes.io/projected/5c4de614-4360-4e42-bbe2-2d5f60c123b3-kube-api-access-s648q\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.918151 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.918118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-tmp-dir\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.918151 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.918153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c4de614-4360-4e42-bbe2-2d5f60c123b3-tls-certs\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.918391 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.918178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-home\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.918391 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.918318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-dshm\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.918508 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.918459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-model-cache\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.918574 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.918514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s648q\" (UniqueName: \"kubernetes.io/projected/5c4de614-4360-4e42-bbe2-2d5f60c123b3-kube-api-access-s648q\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.918574 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.918556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-tmp-dir\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.918666 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.918580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-home\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.918729 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.918713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-model-cache\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.920440 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.920417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-dshm\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.920721 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.920702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c4de614-4360-4e42-bbe2-2d5f60c123b3-tls-certs\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:37.927761 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:37.927738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s648q\" (UniqueName: \"kubernetes.io/projected/5c4de614-4360-4e42-bbe2-2d5f60c123b3-kube-api-access-s648q\") pod \"rate-test-model-59dc59-kserve-74795c8b67-xchq5\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:38.018079 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:38.017964    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:38.291045 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:38.291019    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"]
Sep 10 14:43:38.293290 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:43:38.293265    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c4de614_4360_4e42_bbe2_2d5f60c123b3.slice/crio-b629820d72a5b22dd4af3f0a0626613195a600bdddef496547bc45a6f4732c8b WatchSource:0}: Error finding container b629820d72a5b22dd4af3f0a0626613195a600bdddef496547bc45a6f4732c8b: Status 404 returned error can't find the container with id b629820d72a5b22dd4af3f0a0626613195a600bdddef496547bc45a6f4732c8b
Sep 10 14:43:39.277642 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:39.277596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5" event={"ID":"5c4de614-4360-4e42-bbe2-2d5f60c123b3","Type":"ContainerStarted","Data":"9e07a4016403906a9f0a230b3bfb93c08e1df75c9123349b0d2fecd48cbc6df0"}
Sep 10 14:43:39.277642 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:39.277645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5" event={"ID":"5c4de614-4360-4e42-bbe2-2d5f60c123b3","Type":"ContainerStarted","Data":"b629820d72a5b22dd4af3f0a0626613195a600bdddef496547bc45a6f4732c8b"}
Sep 10 14:43:39.296451 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:39.296402    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5" podStartSLOduration=2.296386845 podStartE2EDuration="2.296386845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:39.293852855 +0000 UTC m=+1437.485381053" watchObservedRunningTime="2026-09-10 14:43:39.296386845 +0000 UTC m=+1437.487915042"
Sep 10 14:43:44.907227 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:44.907192    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"]
Sep 10 14:43:44.907786 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:44.907560    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8" podUID="a473c6ab-d2c9-411f-a3b7-03ec0962caac" containerName="maas-api" containerID="cri-o://dc65e44c745884656abfc16736f4f019dcfa86092d54e4a71d2f15fca78a4581" gracePeriod=30
Sep 10 14:43:45.154273 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.154245    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:43:45.290718 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.290633    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a473c6ab-d2c9-411f-a3b7-03ec0962caac-openshift-service-ca\") pod \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") "
Sep 10 14:43:45.290718 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.290681    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tqpn\" (UniqueName: \"kubernetes.io/projected/a473c6ab-d2c9-411f-a3b7-03ec0962caac-kube-api-access-9tqpn\") pod \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") "
Sep 10 14:43:45.290909 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.290725    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a473c6ab-d2c9-411f-a3b7-03ec0962caac-maas-api-tls\") pod \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") "
Sep 10 14:43:45.290909 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.290769    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a473c6ab-d2c9-411f-a3b7-03ec0962caac-metrics-certs\") pod \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\" (UID: \"a473c6ab-d2c9-411f-a3b7-03ec0962caac\") "
Sep 10 14:43:45.291094 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.291066    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a473c6ab-d2c9-411f-a3b7-03ec0962caac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a473c6ab-d2c9-411f-a3b7-03ec0962caac" (UID: "a473c6ab-d2c9-411f-a3b7-03ec0962caac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:43:45.292957 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.292922    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a473c6ab-d2c9-411f-a3b7-03ec0962caac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a473c6ab-d2c9-411f-a3b7-03ec0962caac" (UID: "a473c6ab-d2c9-411f-a3b7-03ec0962caac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:43:45.293109 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.292985    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a473c6ab-d2c9-411f-a3b7-03ec0962caac-kube-api-access-9tqpn" (OuterVolumeSpecName: "kube-api-access-9tqpn") pod "a473c6ab-d2c9-411f-a3b7-03ec0962caac" (UID: "a473c6ab-d2c9-411f-a3b7-03ec0962caac"). InnerVolumeSpecName "kube-api-access-9tqpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:43:45.293109 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.293033    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a473c6ab-d2c9-411f-a3b7-03ec0962caac-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a473c6ab-d2c9-411f-a3b7-03ec0962caac" (UID: "a473c6ab-d2c9-411f-a3b7-03ec0962caac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:43:45.312383 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.312356    2580 generic.go:356] "Generic (PLEG): container finished" podID="a473c6ab-d2c9-411f-a3b7-03ec0962caac" containerID="dc65e44c745884656abfc16736f4f019dcfa86092d54e4a71d2f15fca78a4581" exitCode=0
Sep 10 14:43:45.312518 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.312396    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"
Sep 10 14:43:45.312518 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.312439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8" event={"ID":"a473c6ab-d2c9-411f-a3b7-03ec0962caac","Type":"ContainerDied","Data":"dc65e44c745884656abfc16736f4f019dcfa86092d54e4a71d2f15fca78a4581"}
Sep 10 14:43:45.312518 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.312490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8" event={"ID":"a473c6ab-d2c9-411f-a3b7-03ec0962caac","Type":"ContainerDied","Data":"56d3baeee50796f6aa0cb0a16cab4e1b0a1d68f9b9dc398aa18b3b53654c26cc"}
Sep 10 14:43:45.312518 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.312511    2580 scope.go:117] "RemoveContainer" containerID="dc65e44c745884656abfc16736f4f019dcfa86092d54e4a71d2f15fca78a4581"
Sep 10 14:43:45.322115 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.322099    2580 scope.go:117] "RemoveContainer" containerID="dc65e44c745884656abfc16736f4f019dcfa86092d54e4a71d2f15fca78a4581"
Sep 10 14:43:45.322388 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:43:45.322366    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc65e44c745884656abfc16736f4f019dcfa86092d54e4a71d2f15fca78a4581\": container with ID starting with dc65e44c745884656abfc16736f4f019dcfa86092d54e4a71d2f15fca78a4581 not found: ID does not exist" containerID="dc65e44c745884656abfc16736f4f019dcfa86092d54e4a71d2f15fca78a4581"
Sep 10 14:43:45.322445 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.322396    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc65e44c745884656abfc16736f4f019dcfa86092d54e4a71d2f15fca78a4581"} err="failed to get container status \"dc65e44c745884656abfc16736f4f019dcfa86092d54e4a71d2f15fca78a4581\": rpc error: code = NotFound desc = could not find container \"dc65e44c745884656abfc16736f4f019dcfa86092d54e4a71d2f15fca78a4581\": container with ID starting with dc65e44c745884656abfc16736f4f019dcfa86092d54e4a71d2f15fca78a4581 not found: ID does not exist"
Sep 10 14:43:45.333684 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.333659    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"]
Sep 10 14:43:45.337735 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.337713    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fc06d4-cdc8f844c-hv8x8"]
Sep 10 14:43:45.392491 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.392463    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a473c6ab-d2c9-411f-a3b7-03ec0962caac-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:43:45.392491 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.392487    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a473c6ab-d2c9-411f-a3b7-03ec0962caac-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:43:45.392491 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.392498    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tqpn\" (UniqueName: \"kubernetes.io/projected/a473c6ab-d2c9-411f-a3b7-03ec0962caac-kube-api-access-9tqpn\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:43:45.392694 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:45.392508    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a473c6ab-d2c9-411f-a3b7-03ec0962caac-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:43:46.309934 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:46.309902    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a473c6ab-d2c9-411f-a3b7-03ec0962caac" path="/var/lib/kubelet/pods/a473c6ab-d2c9-411f-a3b7-03ec0962caac/volumes"
Sep 10 14:43:48.018309 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:48.018278    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:48.018692 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:48.018326    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:48.030910 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:48.030884    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:43:48.341496 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:43:48.341410    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:44:26.029680 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:26.029647    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"]
Sep 10 14:44:27.107594 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:27.107552    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"]
Sep 10 14:44:32.312145 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.312117    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"]
Sep 10 14:44:32.312604 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.312475    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a473c6ab-d2c9-411f-a3b7-03ec0962caac" containerName="maas-api"
Sep 10 14:44:32.312604 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.312485    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a473c6ab-d2c9-411f-a3b7-03ec0962caac" containerName="maas-api"
Sep 10 14:44:32.312604 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.312553    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a473c6ab-d2c9-411f-a3b7-03ec0962caac" containerName="maas-api"
Sep 10 14:44:32.316018 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.315964    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:32.318612 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.318506    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-42514d-serving-cert\""
Sep 10 14:44:32.319232 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.319203    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"]
Sep 10 14:44:32.447452 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.447406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07faef23-22cf-4000-8966-eef38dbd15a8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:32.447452 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.447456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58dk6\" (UniqueName: \"kubernetes.io/projected/07faef23-22cf-4000-8966-eef38dbd15a8-kube-api-access-58dk6\") pod \"maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:32.447709 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.447565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07faef23-22cf-4000-8966-eef38dbd15a8-metrics-certs\") pod \"maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:32.447709 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.447669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07faef23-22cf-4000-8966-eef38dbd15a8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:32.549061 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.549026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07faef23-22cf-4000-8966-eef38dbd15a8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:32.549260 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.549088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07faef23-22cf-4000-8966-eef38dbd15a8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:32.549260 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.549112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58dk6\" (UniqueName: \"kubernetes.io/projected/07faef23-22cf-4000-8966-eef38dbd15a8-kube-api-access-58dk6\") pod \"maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:32.549260 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.549156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07faef23-22cf-4000-8966-eef38dbd15a8-metrics-certs\") pod \"maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:32.549260 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:44:32.549188    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-serving-cert: secret "maas-api-e2e-ipp-a-42514d-serving-cert" not found
Sep 10 14:44:32.549477 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:44:32.549270    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07faef23-22cf-4000-8966-eef38dbd15a8-maas-api-tls podName:07faef23-22cf-4000-8966-eef38dbd15a8 nodeName:}" failed. No retries permitted until 2026-09-10 14:44:33.049249587 +0000 UTC m=+1491.240777763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07faef23-22cf-4000-8966-eef38dbd15a8-maas-api-tls") pod "maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp" (UID: "07faef23-22cf-4000-8966-eef38dbd15a8") : secret "maas-api-e2e-ipp-a-42514d-serving-cert" not found
Sep 10 14:44:32.549701 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.549681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07faef23-22cf-4000-8966-eef38dbd15a8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:32.551600 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.551580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07faef23-22cf-4000-8966-eef38dbd15a8-metrics-certs\") pod \"maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:32.568343 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:32.568271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58dk6\" (UniqueName: \"kubernetes.io/projected/07faef23-22cf-4000-8966-eef38dbd15a8-kube-api-access-58dk6\") pod \"maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:33.053401 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:33.053359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07faef23-22cf-4000-8966-eef38dbd15a8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:33.055908 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:33.055880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07faef23-22cf-4000-8966-eef38dbd15a8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:33.228569 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:33.228533    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:33.515508 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:33.515465    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"]
Sep 10 14:44:33.517171 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:44:33.517140    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07faef23_22cf_4000_8966_eef38dbd15a8.slice/crio-fba38c8844d5d65a97517a5a1bd026d1b228f4c1c9fd25ef82d5f600aff9351b WatchSource:0}: Error finding container fba38c8844d5d65a97517a5a1bd026d1b228f4c1c9fd25ef82d5f600aff9351b: Status 404 returned error can't find the container with id fba38c8844d5d65a97517a5a1bd026d1b228f4c1c9fd25ef82d5f600aff9351b
Sep 10 14:44:33.586995 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:33.586940    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp" event={"ID":"07faef23-22cf-4000-8966-eef38dbd15a8","Type":"ContainerStarted","Data":"fba38c8844d5d65a97517a5a1bd026d1b228f4c1c9fd25ef82d5f600aff9351b"}
Sep 10 14:44:34.593468 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:34.593433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp" event={"ID":"07faef23-22cf-4000-8966-eef38dbd15a8","Type":"ContainerStarted","Data":"f512bb8dbd7fb35350d4aad3d3291463822e970e7ee2edf063dd1129389a6c25"}
Sep 10 14:44:34.593827 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:34.593566    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:34.610922 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:34.610864    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp" podStartSLOduration=2.2294679410000002 podStartE2EDuration="2.610848227s" totalImagesPullingTime="381.380286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:44:34.609790223 +0000 UTC m=+1492.801318422" watchObservedRunningTime="2026-09-10 14:44:34.610848227 +0000 UTC m=+1492.802376424"
Sep 10 14:44:40.605919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:40.605884    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:44:41.049780 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.049738    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5" podUID="1e14836f-1145-41ee-867a-54b7afe8811b" containerName="main" containerID="cri-o://3bd608381b9c18963f7d47062b92b358e99c9075eac54daa3157912e2eb3dfbf" gracePeriod=45
Sep 10 14:44:41.300785 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.300725    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:44:41.330016 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.329951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-dshm\") pod \"1e14836f-1145-41ee-867a-54b7afe8811b\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") "
Sep 10 14:44:41.330211 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.330069    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-model-cache\") pod \"1e14836f-1145-41ee-867a-54b7afe8811b\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") "
Sep 10 14:44:41.330211 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.330107    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt97p\" (UniqueName: \"kubernetes.io/projected/1e14836f-1145-41ee-867a-54b7afe8811b-kube-api-access-kt97p\") pod \"1e14836f-1145-41ee-867a-54b7afe8811b\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") "
Sep 10 14:44:41.330211 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.330132    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-tmp-dir\") pod \"1e14836f-1145-41ee-867a-54b7afe8811b\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") "
Sep 10 14:44:41.330211 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.330181    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e14836f-1145-41ee-867a-54b7afe8811b-tls-certs\") pod \"1e14836f-1145-41ee-867a-54b7afe8811b\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") "
Sep 10 14:44:41.330453 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.330234    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-home\") pod \"1e14836f-1145-41ee-867a-54b7afe8811b\" (UID: \"1e14836f-1145-41ee-867a-54b7afe8811b\") "
Sep 10 14:44:41.330453 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.330369    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e14836f-1145-41ee-867a-54b7afe8811b" (UID: "1e14836f-1145-41ee-867a-54b7afe8811b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:44:41.330453 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.330409    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e14836f-1145-41ee-867a-54b7afe8811b" (UID: "1e14836f-1145-41ee-867a-54b7afe8811b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:44:41.330620 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.330565    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-home" (OuterVolumeSpecName: "home") pod "1e14836f-1145-41ee-867a-54b7afe8811b" (UID: "1e14836f-1145-41ee-867a-54b7afe8811b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:44:41.330679 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.330633    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:44:41.330679 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.330655    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:44:41.330679 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.330674    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:44:41.332442 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.332412    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e14836f-1145-41ee-867a-54b7afe8811b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e14836f-1145-41ee-867a-54b7afe8811b" (UID: "1e14836f-1145-41ee-867a-54b7afe8811b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:44:41.332554 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.332455    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e14836f-1145-41ee-867a-54b7afe8811b-kube-api-access-kt97p" (OuterVolumeSpecName: "kube-api-access-kt97p") pod "1e14836f-1145-41ee-867a-54b7afe8811b" (UID: "1e14836f-1145-41ee-867a-54b7afe8811b"). InnerVolumeSpecName "kube-api-access-kt97p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:44:41.332554 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.332502    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-dshm" (OuterVolumeSpecName: "dshm") pod "1e14836f-1145-41ee-867a-54b7afe8811b" (UID: "1e14836f-1145-41ee-867a-54b7afe8811b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:44:41.432211 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.432175    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e14836f-1145-41ee-867a-54b7afe8811b-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:44:41.432211 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.432205    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt97p\" (UniqueName: \"kubernetes.io/projected/1e14836f-1145-41ee-867a-54b7afe8811b-kube-api-access-kt97p\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:44:41.432211 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.432215    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e14836f-1145-41ee-867a-54b7afe8811b-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:44:41.643053 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.643015    2580 generic.go:356] "Generic (PLEG): container finished" podID="1e14836f-1145-41ee-867a-54b7afe8811b" containerID="3bd608381b9c18963f7d47062b92b358e99c9075eac54daa3157912e2eb3dfbf" exitCode=0
Sep 10 14:44:41.643523 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.643060    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"
Sep 10 14:44:41.643523 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.643096    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5" event={"ID":"1e14836f-1145-41ee-867a-54b7afe8811b","Type":"ContainerDied","Data":"3bd608381b9c18963f7d47062b92b358e99c9075eac54daa3157912e2eb3dfbf"}
Sep 10 14:44:41.643523 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.643137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5" event={"ID":"1e14836f-1145-41ee-867a-54b7afe8811b","Type":"ContainerDied","Data":"1ec9491724375ea53e928dc427953a8aa00ec69d69c186c14a8fbd6e010b5d29"}
Sep 10 14:44:41.643523 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.643153    2580 scope.go:117] "RemoveContainer" containerID="3bd608381b9c18963f7d47062b92b358e99c9075eac54daa3157912e2eb3dfbf"
Sep 10 14:44:41.653309 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.653293    2580 scope.go:117] "RemoveContainer" containerID="3bd608381b9c18963f7d47062b92b358e99c9075eac54daa3157912e2eb3dfbf"
Sep 10 14:44:41.653614 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:44:41.653595    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bd608381b9c18963f7d47062b92b358e99c9075eac54daa3157912e2eb3dfbf\": container with ID starting with 3bd608381b9c18963f7d47062b92b358e99c9075eac54daa3157912e2eb3dfbf not found: ID does not exist" containerID="3bd608381b9c18963f7d47062b92b358e99c9075eac54daa3157912e2eb3dfbf"
Sep 10 14:44:41.653666 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.653625    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bd608381b9c18963f7d47062b92b358e99c9075eac54daa3157912e2eb3dfbf"} err="failed to get container status \"3bd608381b9c18963f7d47062b92b358e99c9075eac54daa3157912e2eb3dfbf\": rpc error: code = NotFound desc = could not find container \"3bd608381b9c18963f7d47062b92b358e99c9075eac54daa3157912e2eb3dfbf\": container with ID starting with 3bd608381b9c18963f7d47062b92b358e99c9075eac54daa3157912e2eb3dfbf not found: ID does not exist"
Sep 10 14:44:41.665082 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.665057    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"]
Sep 10 14:44:41.669008 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:41.668985    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7bf16f/rate-test-model-7bf16f-kserve-5bbd4c9475-hmxx5"]
Sep 10 14:44:42.128578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.128491    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5" podUID="5c4de614-4360-4e42-bbe2-2d5f60c123b3" containerName="main" containerID="cri-o://9e07a4016403906a9f0a230b3bfb93c08e1df75c9123349b0d2fecd48cbc6df0" gracePeriod=45
Sep 10 14:44:42.311164 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.311132    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e14836f-1145-41ee-867a-54b7afe8811b" path="/var/lib/kubelet/pods/1e14836f-1145-41ee-867a-54b7afe8811b/volumes"
Sep 10 14:44:42.381776 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.381753    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:44:42.389472 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.389449    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:44:42.391339 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.391321    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:44:42.441801 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.441768    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-model-cache\") pod \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") "
Sep 10 14:44:42.442006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.441817    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-dshm\") pod \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") "
Sep 10 14:44:42.442006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.441851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-tmp-dir\") pod \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") "
Sep 10 14:44:42.442006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.441880    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c4de614-4360-4e42-bbe2-2d5f60c123b3-tls-certs\") pod \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") "
Sep 10 14:44:42.442006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.441894    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-home\") pod \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") "
Sep 10 14:44:42.442239 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.442023    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s648q\" (UniqueName: \"kubernetes.io/projected/5c4de614-4360-4e42-bbe2-2d5f60c123b3-kube-api-access-s648q\") pod \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\" (UID: \"5c4de614-4360-4e42-bbe2-2d5f60c123b3\") "
Sep 10 14:44:42.442239 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.442122    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c4de614-4360-4e42-bbe2-2d5f60c123b3" (UID: "5c4de614-4360-4e42-bbe2-2d5f60c123b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:44:42.442239 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.442157    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c4de614-4360-4e42-bbe2-2d5f60c123b3" (UID: "5c4de614-4360-4e42-bbe2-2d5f60c123b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:44:42.442382 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.442354    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-home" (OuterVolumeSpecName: "home") pod "5c4de614-4360-4e42-bbe2-2d5f60c123b3" (UID: "5c4de614-4360-4e42-bbe2-2d5f60c123b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:44:42.442382 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.442374    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:44:42.442468 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.442393    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:44:42.444040 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.444015    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-dshm" (OuterVolumeSpecName: "dshm") pod "5c4de614-4360-4e42-bbe2-2d5f60c123b3" (UID: "5c4de614-4360-4e42-bbe2-2d5f60c123b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:44:42.444157 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.444067    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c4de614-4360-4e42-bbe2-2d5f60c123b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c4de614-4360-4e42-bbe2-2d5f60c123b3" (UID: "5c4de614-4360-4e42-bbe2-2d5f60c123b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:44:42.444222 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.444201    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4de614-4360-4e42-bbe2-2d5f60c123b3-kube-api-access-s648q" (OuterVolumeSpecName: "kube-api-access-s648q") pod "5c4de614-4360-4e42-bbe2-2d5f60c123b3" (UID: "5c4de614-4360-4e42-bbe2-2d5f60c123b3"). InnerVolumeSpecName "kube-api-access-s648q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:44:42.543508 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.543470    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c4de614-4360-4e42-bbe2-2d5f60c123b3-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:44:42.543508 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.543502    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:44:42.543508 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.543512    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s648q\" (UniqueName: \"kubernetes.io/projected/5c4de614-4360-4e42-bbe2-2d5f60c123b3-kube-api-access-s648q\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:44:42.543755 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.543521    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c4de614-4360-4e42-bbe2-2d5f60c123b3-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:44:42.650352 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.650273    2580 generic.go:356] "Generic (PLEG): container finished" podID="5c4de614-4360-4e42-bbe2-2d5f60c123b3" containerID="9e07a4016403906a9f0a230b3bfb93c08e1df75c9123349b0d2fecd48cbc6df0" exitCode=0
Sep 10 14:44:42.650352 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.650330    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"
Sep 10 14:44:42.650803 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.650355    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5" event={"ID":"5c4de614-4360-4e42-bbe2-2d5f60c123b3","Type":"ContainerDied","Data":"9e07a4016403906a9f0a230b3bfb93c08e1df75c9123349b0d2fecd48cbc6df0"}
Sep 10 14:44:42.650803 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.650399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5" event={"ID":"5c4de614-4360-4e42-bbe2-2d5f60c123b3","Type":"ContainerDied","Data":"b629820d72a5b22dd4af3f0a0626613195a600bdddef496547bc45a6f4732c8b"}
Sep 10 14:44:42.650803 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.650421    2580 scope.go:117] "RemoveContainer" containerID="9e07a4016403906a9f0a230b3bfb93c08e1df75c9123349b0d2fecd48cbc6df0"
Sep 10 14:44:42.661390 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.661359    2580 scope.go:117] "RemoveContainer" containerID="9e07a4016403906a9f0a230b3bfb93c08e1df75c9123349b0d2fecd48cbc6df0"
Sep 10 14:44:42.661765 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:44:42.661733    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e07a4016403906a9f0a230b3bfb93c08e1df75c9123349b0d2fecd48cbc6df0\": container with ID starting with 9e07a4016403906a9f0a230b3bfb93c08e1df75c9123349b0d2fecd48cbc6df0 not found: ID does not exist" containerID="9e07a4016403906a9f0a230b3bfb93c08e1df75c9123349b0d2fecd48cbc6df0"
Sep 10 14:44:42.661915 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.661890    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e07a4016403906a9f0a230b3bfb93c08e1df75c9123349b0d2fecd48cbc6df0"} err="failed to get container status \"9e07a4016403906a9f0a230b3bfb93c08e1df75c9123349b0d2fecd48cbc6df0\": rpc error: code = NotFound desc = could not find container \"9e07a4016403906a9f0a230b3bfb93c08e1df75c9123349b0d2fecd48cbc6df0\": container with ID starting with 9e07a4016403906a9f0a230b3bfb93c08e1df75c9123349b0d2fecd48cbc6df0 not found: ID does not exist"
Sep 10 14:44:42.679036 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.679005    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"]
Sep 10 14:44:42.683149 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:42.683126    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/rate-test-model-59dc59-kserve-74795c8b67-xchq5"]
Sep 10 14:44:44.310133 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:44:44.310098    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c4de614-4360-4e42-bbe2-2d5f60c123b3" path="/var/lib/kubelet/pods/5c4de614-4360-4e42-bbe2-2d5f60c123b3/volumes"
Sep 10 14:45:01.508861 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.508826    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"]
Sep 10 14:45:01.509286 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.509230    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4de614-4360-4e42-bbe2-2d5f60c123b3" containerName="main"
Sep 10 14:45:01.509286 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.509242    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4de614-4360-4e42-bbe2-2d5f60c123b3" containerName="main"
Sep 10 14:45:01.509286 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.509256    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e14836f-1145-41ee-867a-54b7afe8811b" containerName="main"
Sep 10 14:45:01.509286 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.509261    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e14836f-1145-41ee-867a-54b7afe8811b" containerName="main"
Sep 10 14:45:01.509443 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.509328    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c4de614-4360-4e42-bbe2-2d5f60c123b3" containerName="main"
Sep 10 14:45:01.509443 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.509339    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e14836f-1145-41ee-867a-54b7afe8811b" containerName="main"
Sep 10 14:45:01.513615 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.513595    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:01.516132 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.516107    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5c43af-serving-cert\""
Sep 10 14:45:01.524042 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.524019    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"]
Sep 10 14:45:01.622789 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.622754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b32f788-cd11-4988-b457-eff711e71fef-metrics-certs\") pod \"maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:01.623006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.622797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zb4g\" (UniqueName: \"kubernetes.io/projected/0b32f788-cd11-4988-b457-eff711e71fef-kube-api-access-2zb4g\") pod \"maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:01.623006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.622875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b32f788-cd11-4988-b457-eff711e71fef-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:01.623006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.622948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b32f788-cd11-4988-b457-eff711e71fef-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:01.723591 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.723554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b32f788-cd11-4988-b457-eff711e71fef-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:01.723781 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.723678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b32f788-cd11-4988-b457-eff711e71fef-metrics-certs\") pod \"maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:01.723781 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.723719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zb4g\" (UniqueName: \"kubernetes.io/projected/0b32f788-cd11-4988-b457-eff711e71fef-kube-api-access-2zb4g\") pod \"maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:01.723781 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.723768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b32f788-cd11-4988-b457-eff711e71fef-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:01.724527 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.724499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b32f788-cd11-4988-b457-eff711e71fef-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:01.726335 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.726311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b32f788-cd11-4988-b457-eff711e71fef-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:01.726444 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.726311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b32f788-cd11-4988-b457-eff711e71fef-metrics-certs\") pod \"maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:01.731850 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.731827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zb4g\" (UniqueName: \"kubernetes.io/projected/0b32f788-cd11-4988-b457-eff711e71fef-kube-api-access-2zb4g\") pod \"maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:01.825427 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:01.825341    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:02.100554 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:02.100525    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"]
Sep 10 14:45:02.101620 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:45:02.101573    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b32f788_cd11_4988_b457_eff711e71fef.slice/crio-2603e7f5f6b84b28bd7967062ad3914448a70bd45fb2265bb5a628f5565d21d6 WatchSource:0}: Error finding container 2603e7f5f6b84b28bd7967062ad3914448a70bd45fb2265bb5a628f5565d21d6: Status 404 returned error can't find the container with id 2603e7f5f6b84b28bd7967062ad3914448a70bd45fb2265bb5a628f5565d21d6
Sep 10 14:45:02.763909 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:02.763872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr" event={"ID":"0b32f788-cd11-4988-b457-eff711e71fef","Type":"ContainerStarted","Data":"dd6939e164c7a3e0bb279d22a877726511f2ce9b2046bdea6b8c6ba094cc8e27"}
Sep 10 14:45:02.764323 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:02.764092    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:02.764323 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:02.764129    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr" event={"ID":"0b32f788-cd11-4988-b457-eff711e71fef","Type":"ContainerStarted","Data":"2603e7f5f6b84b28bd7967062ad3914448a70bd45fb2265bb5a628f5565d21d6"}
Sep 10 14:45:02.785666 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:02.785611    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr" podStartSLOduration=1.292066679 podStartE2EDuration="1.785593311s" totalImagesPullingTime="493.526632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:45:02.784452573 +0000 UTC m=+1520.975980772" watchObservedRunningTime="2026-09-10 14:45:02.785593311 +0000 UTC m=+1520.977121508"
Sep 10 14:45:08.774604 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:08.774575    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:45:36.207813 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.207779    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"]
Sep 10 14:45:36.211523 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.211500    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.217258 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.214447    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-42514d\"/\"ipp-route-42514d-kserve-self-signed-certs\""
Sep 10 14:45:36.217258 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.215459    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-42514d\"/\"default-dockercfg-m6k2f\""
Sep 10 14:45:36.217258 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.215734    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-42514d\"/\"kube-root-ca.crt\""
Sep 10 14:45:36.217258 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.216037    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-42514d\"/\"openshift-service-ca.crt\""
Sep 10 14:45:36.222877 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.222850    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"]
Sep 10 14:45:36.344227 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.344194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-tmp-dir\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.344422 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.344244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-tls-certs\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.344422 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.344362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-dshm\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.344526 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.344461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqvlt\" (UniqueName: \"kubernetes.io/projected/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-kube-api-access-pqvlt\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.344526 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.344487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-model-cache\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.344613 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.344538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-home\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.445219 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.445178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqvlt\" (UniqueName: \"kubernetes.io/projected/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-kube-api-access-pqvlt\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.445219 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.445220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-model-cache\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.445474 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.445246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-home\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.445474 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.445287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-tmp-dir\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.445474 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.445317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-tls-certs\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.445622 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.445481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-dshm\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.445702 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.445680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-model-cache\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.445742 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.445717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-home\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.445778 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.445736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-tmp-dir\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.447601 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.447578    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-dshm\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.447754 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.447738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-tls-certs\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.453507 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.453483    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqvlt\" (UniqueName: \"kubernetes.io/projected/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-kube-api-access-pqvlt\") pod \"ipp-route-42514d-kserve-5ff45cd4cf-9gbmc\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") " pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.525424 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.525314    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:36.801341 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.801314    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"]
Sep 10 14:45:36.802071 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:45:36.802044    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3419a59_e23a_4f3d_8f74_5ad1f5104eb6.slice/crio-17b17ce2ae6dd193c7ec98ade844a39c4f4419e690bc19874ebaa87fcab1b8ef WatchSource:0}: Error finding container 17b17ce2ae6dd193c7ec98ade844a39c4f4419e690bc19874ebaa87fcab1b8ef: Status 404 returned error can't find the container with id 17b17ce2ae6dd193c7ec98ade844a39c4f4419e690bc19874ebaa87fcab1b8ef
Sep 10 14:45:36.973753 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.973717    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc" event={"ID":"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6","Type":"ContainerStarted","Data":"585e1865f3d97af24699f87b73b2bc75b5ccb14b5c366786b708fbae07e698d8"}
Sep 10 14:45:36.973935 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.973768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc" event={"ID":"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6","Type":"ContainerStarted","Data":"17b17ce2ae6dd193c7ec98ade844a39c4f4419e690bc19874ebaa87fcab1b8ef"}
Sep 10 14:45:36.993799 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:36.993742    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc" podStartSLOduration=0.993726322 podStartE2EDuration="993.726322ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:45:36.991054631 +0000 UTC m=+1555.182582829" watchObservedRunningTime="2026-09-10 14:45:36.993726322 +0000 UTC m=+1555.185254520"
Sep 10 14:45:46.525691 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:46.525653    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:46.526193 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:46.525708    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:46.538075 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:46.538052    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:45:47.041269 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:45:47.041232    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:46:06.105984 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.105936    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"]
Sep 10 14:46:06.109794 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.109774    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:06.113790 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.113766    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-268239-serving-cert\""
Sep 10 14:46:06.130368 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.130343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5074be9b-503b-47dd-a59e-0e6632cb30e4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:06.130495 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.130413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5074be9b-503b-47dd-a59e-0e6632cb30e4-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:06.130495 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.130443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwx7s\" (UniqueName: \"kubernetes.io/projected/5074be9b-503b-47dd-a59e-0e6632cb30e4-kube-api-access-qwx7s\") pod \"maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:06.130495 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.130466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5074be9b-503b-47dd-a59e-0e6632cb30e4-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:06.222576 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.222538    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"]
Sep 10 14:46:06.231546 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.231511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5074be9b-503b-47dd-a59e-0e6632cb30e4-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:06.231740 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.231625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5074be9b-503b-47dd-a59e-0e6632cb30e4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:06.231740 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.231700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5074be9b-503b-47dd-a59e-0e6632cb30e4-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:06.231740 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.231729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwx7s\" (UniqueName: \"kubernetes.io/projected/5074be9b-503b-47dd-a59e-0e6632cb30e4-kube-api-access-qwx7s\") pod \"maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:06.231921 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:46:06.231781    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-serving-cert: secret "maas-api-e2e-ipp-cleanup-268239-serving-cert" not found
Sep 10 14:46:06.231921 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:46:06.231873    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5074be9b-503b-47dd-a59e-0e6632cb30e4-maas-api-tls podName:5074be9b-503b-47dd-a59e-0e6632cb30e4 nodeName:}" failed. No retries permitted until 2026-09-10 14:46:06.731850548 +0000 UTC m=+1584.923378741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5074be9b-503b-47dd-a59e-0e6632cb30e4-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt" (UID: "5074be9b-503b-47dd-a59e-0e6632cb30e4") : secret "maas-api-e2e-ipp-cleanup-268239-serving-cert" not found
Sep 10 14:46:06.232423 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.232393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5074be9b-503b-47dd-a59e-0e6632cb30e4-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:06.234292 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.234270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5074be9b-503b-47dd-a59e-0e6632cb30e4-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:06.257159 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.257136    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwx7s\" (UniqueName: \"kubernetes.io/projected/5074be9b-503b-47dd-a59e-0e6632cb30e4-kube-api-access-qwx7s\") pod \"maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:06.736933 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.736897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5074be9b-503b-47dd-a59e-0e6632cb30e4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:06.739400 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:06.739378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5074be9b-503b-47dd-a59e-0e6632cb30e4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:07.020042 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:07.019939    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:07.298700 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:07.298667    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"]
Sep 10 14:46:07.300721 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:46:07.300695    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5074be9b_503b_47dd_a59e_0e6632cb30e4.slice/crio-48ad8a4df2ad9088ca94b45089b87081c7fb6d13e559f1c2dede38e75d270c43 WatchSource:0}: Error finding container 48ad8a4df2ad9088ca94b45089b87081c7fb6d13e559f1c2dede38e75d270c43: Status 404 returned error can't find the container with id 48ad8a4df2ad9088ca94b45089b87081c7fb6d13e559f1c2dede38e75d270c43
Sep 10 14:46:08.145567 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:08.145532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt" event={"ID":"5074be9b-503b-47dd-a59e-0e6632cb30e4","Type":"ContainerStarted","Data":"c5bd0e89d2d348a1ac8378b8f694affd97b883ca5c9055e948339404e4f76c2c"}
Sep 10 14:46:08.145763 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:08.145595    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt" event={"ID":"5074be9b-503b-47dd-a59e-0e6632cb30e4","Type":"ContainerStarted","Data":"48ad8a4df2ad9088ca94b45089b87081c7fb6d13e559f1c2dede38e75d270c43"}
Sep 10 14:46:08.145763 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:08.145735    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:08.162732 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:08.162689    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt" podStartSLOduration=1.605916638 podStartE2EDuration="2.162675443s" totalImagesPullingTime="556.758805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:46:08.161074758 +0000 UTC m=+1586.352602956" watchObservedRunningTime="2026-09-10 14:46:08.162675443 +0000 UTC m=+1586.354203641"
Sep 10 14:46:14.157141 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:14.157112    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:36.355356 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:36.355276    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"]
Sep 10 14:46:36.355907 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:36.355482    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt" podUID="5074be9b-503b-47dd-a59e-0e6632cb30e4" containerName="maas-api" containerID="cri-o://c5bd0e89d2d348a1ac8378b8f694affd97b883ca5c9055e948339404e4f76c2c" gracePeriod=30
Sep 10 14:46:36.610122 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:36.610057    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:36.730027 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:36.729964    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwx7s\" (UniqueName: \"kubernetes.io/projected/5074be9b-503b-47dd-a59e-0e6632cb30e4-kube-api-access-qwx7s\") pod \"5074be9b-503b-47dd-a59e-0e6632cb30e4\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") "
Sep 10 14:46:36.730192 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:36.730039    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5074be9b-503b-47dd-a59e-0e6632cb30e4-maas-api-tls\") pod \"5074be9b-503b-47dd-a59e-0e6632cb30e4\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") "
Sep 10 14:46:36.730192 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:36.730129    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5074be9b-503b-47dd-a59e-0e6632cb30e4-metrics-certs\") pod \"5074be9b-503b-47dd-a59e-0e6632cb30e4\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") "
Sep 10 14:46:36.730278 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:36.730258    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5074be9b-503b-47dd-a59e-0e6632cb30e4-openshift-service-ca\") pod \"5074be9b-503b-47dd-a59e-0e6632cb30e4\" (UID: \"5074be9b-503b-47dd-a59e-0e6632cb30e4\") "
Sep 10 14:46:36.730582 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:36.730550    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5074be9b-503b-47dd-a59e-0e6632cb30e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5074be9b-503b-47dd-a59e-0e6632cb30e4" (UID: "5074be9b-503b-47dd-a59e-0e6632cb30e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:46:36.732105 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:36.732080    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5074be9b-503b-47dd-a59e-0e6632cb30e4-kube-api-access-qwx7s" (OuterVolumeSpecName: "kube-api-access-qwx7s") pod "5074be9b-503b-47dd-a59e-0e6632cb30e4" (UID: "5074be9b-503b-47dd-a59e-0e6632cb30e4"). InnerVolumeSpecName "kube-api-access-qwx7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:46:36.732610 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:36.732579    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5074be9b-503b-47dd-a59e-0e6632cb30e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5074be9b-503b-47dd-a59e-0e6632cb30e4" (UID: "5074be9b-503b-47dd-a59e-0e6632cb30e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:46:36.732699 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:36.732588    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5074be9b-503b-47dd-a59e-0e6632cb30e4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5074be9b-503b-47dd-a59e-0e6632cb30e4" (UID: "5074be9b-503b-47dd-a59e-0e6632cb30e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:46:36.831293 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:36.831255    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwx7s\" (UniqueName: \"kubernetes.io/projected/5074be9b-503b-47dd-a59e-0e6632cb30e4-kube-api-access-qwx7s\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:46:36.831293 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:36.831295    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5074be9b-503b-47dd-a59e-0e6632cb30e4-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:46:36.831487 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:36.831311    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5074be9b-503b-47dd-a59e-0e6632cb30e4-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:46:36.831487 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:36.831326    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5074be9b-503b-47dd-a59e-0e6632cb30e4-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:46:37.313047 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:37.313013    2580 generic.go:356] "Generic (PLEG): container finished" podID="5074be9b-503b-47dd-a59e-0e6632cb30e4" containerID="c5bd0e89d2d348a1ac8378b8f694affd97b883ca5c9055e948339404e4f76c2c" exitCode=0
Sep 10 14:46:37.313229 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:37.313053    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"
Sep 10 14:46:37.313229 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:37.313057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt" event={"ID":"5074be9b-503b-47dd-a59e-0e6632cb30e4","Type":"ContainerDied","Data":"c5bd0e89d2d348a1ac8378b8f694affd97b883ca5c9055e948339404e4f76c2c"}
Sep 10 14:46:37.313229 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:37.313104    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt" event={"ID":"5074be9b-503b-47dd-a59e-0e6632cb30e4","Type":"ContainerDied","Data":"48ad8a4df2ad9088ca94b45089b87081c7fb6d13e559f1c2dede38e75d270c43"}
Sep 10 14:46:37.313229 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:37.313126    2580 scope.go:117] "RemoveContainer" containerID="c5bd0e89d2d348a1ac8378b8f694affd97b883ca5c9055e948339404e4f76c2c"
Sep 10 14:46:37.323179 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:37.323160    2580 scope.go:117] "RemoveContainer" containerID="c5bd0e89d2d348a1ac8378b8f694affd97b883ca5c9055e948339404e4f76c2c"
Sep 10 14:46:37.323440 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:46:37.323420    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5bd0e89d2d348a1ac8378b8f694affd97b883ca5c9055e948339404e4f76c2c\": container with ID starting with c5bd0e89d2d348a1ac8378b8f694affd97b883ca5c9055e948339404e4f76c2c not found: ID does not exist" containerID="c5bd0e89d2d348a1ac8378b8f694affd97b883ca5c9055e948339404e4f76c2c"
Sep 10 14:46:37.323490 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:37.323449    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5bd0e89d2d348a1ac8378b8f694affd97b883ca5c9055e948339404e4f76c2c"} err="failed to get container status \"c5bd0e89d2d348a1ac8378b8f694affd97b883ca5c9055e948339404e4f76c2c\": rpc error: code = NotFound desc = could not find container \"c5bd0e89d2d348a1ac8378b8f694affd97b883ca5c9055e948339404e4f76c2c\": container with ID starting with c5bd0e89d2d348a1ac8378b8f694affd97b883ca5c9055e948339404e4f76c2c not found: ID does not exist"
Sep 10 14:46:37.338672 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:37.338646    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"]
Sep 10 14:46:37.347026 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:37.347003    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-268239-69b74f794d-znkqt"]
Sep 10 14:46:38.310166 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:46:38.310136    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5074be9b-503b-47dd-a59e-0e6632cb30e4" path="/var/lib/kubelet/pods/5074be9b-503b-47dd-a59e-0e6632cb30e4/volumes"
Sep 10 14:47:02.214893 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.214857    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"]
Sep 10 14:47:02.215471 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.215134    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp" podUID="07faef23-22cf-4000-8966-eef38dbd15a8" containerName="maas-api" containerID="cri-o://f512bb8dbd7fb35350d4aad3d3291463822e970e7ee2edf063dd1129389a6c25" gracePeriod=30
Sep 10 14:47:02.455942 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.455906    2580 generic.go:356] "Generic (PLEG): container finished" podID="07faef23-22cf-4000-8966-eef38dbd15a8" containerID="f512bb8dbd7fb35350d4aad3d3291463822e970e7ee2edf063dd1129389a6c25" exitCode=0
Sep 10 14:47:02.456089 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.455993    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp" event={"ID":"07faef23-22cf-4000-8966-eef38dbd15a8","Type":"ContainerDied","Data":"f512bb8dbd7fb35350d4aad3d3291463822e970e7ee2edf063dd1129389a6c25"}
Sep 10 14:47:02.456089 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.456051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp" event={"ID":"07faef23-22cf-4000-8966-eef38dbd15a8","Type":"ContainerDied","Data":"fba38c8844d5d65a97517a5a1bd026d1b228f4c1c9fd25ef82d5f600aff9351b"}
Sep 10 14:47:02.456089 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.456072    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fba38c8844d5d65a97517a5a1bd026d1b228f4c1c9fd25ef82d5f600aff9351b"
Sep 10 14:47:02.466318 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.466260    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:47:02.562250 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.562217    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07faef23-22cf-4000-8966-eef38dbd15a8-maas-api-tls\") pod \"07faef23-22cf-4000-8966-eef38dbd15a8\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") "
Sep 10 14:47:02.562433 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.562264    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07faef23-22cf-4000-8966-eef38dbd15a8-metrics-certs\") pod \"07faef23-22cf-4000-8966-eef38dbd15a8\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") "
Sep 10 14:47:02.562433 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.562301    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58dk6\" (UniqueName: \"kubernetes.io/projected/07faef23-22cf-4000-8966-eef38dbd15a8-kube-api-access-58dk6\") pod \"07faef23-22cf-4000-8966-eef38dbd15a8\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") "
Sep 10 14:47:02.562433 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.562380    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07faef23-22cf-4000-8966-eef38dbd15a8-openshift-service-ca\") pod \"07faef23-22cf-4000-8966-eef38dbd15a8\" (UID: \"07faef23-22cf-4000-8966-eef38dbd15a8\") "
Sep 10 14:47:02.562887 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.562849    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07faef23-22cf-4000-8966-eef38dbd15a8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07faef23-22cf-4000-8966-eef38dbd15a8" (UID: "07faef23-22cf-4000-8966-eef38dbd15a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:47:02.564502 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.564475    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07faef23-22cf-4000-8966-eef38dbd15a8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07faef23-22cf-4000-8966-eef38dbd15a8" (UID: "07faef23-22cf-4000-8966-eef38dbd15a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:47:02.564502 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.564495    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07faef23-22cf-4000-8966-eef38dbd15a8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "07faef23-22cf-4000-8966-eef38dbd15a8" (UID: "07faef23-22cf-4000-8966-eef38dbd15a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:47:02.564645 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.564548    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07faef23-22cf-4000-8966-eef38dbd15a8-kube-api-access-58dk6" (OuterVolumeSpecName: "kube-api-access-58dk6") pod "07faef23-22cf-4000-8966-eef38dbd15a8" (UID: "07faef23-22cf-4000-8966-eef38dbd15a8"). InnerVolumeSpecName "kube-api-access-58dk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:47:02.663891 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.663842    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07faef23-22cf-4000-8966-eef38dbd15a8-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:47:02.663891 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.663886    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07faef23-22cf-4000-8966-eef38dbd15a8-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:47:02.663891 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.663895    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58dk6\" (UniqueName: \"kubernetes.io/projected/07faef23-22cf-4000-8966-eef38dbd15a8-kube-api-access-58dk6\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:47:02.663891 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:02.663903    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07faef23-22cf-4000-8966-eef38dbd15a8-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:47:03.461627 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:03.461596    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"
Sep 10 14:47:03.485800 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:03.485767    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"]
Sep 10 14:47:03.489440 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:03.489406    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-42514d-7c86b86bb5-hlclp"]
Sep 10 14:47:04.310617 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:04.310583    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07faef23-22cf-4000-8966-eef38dbd15a8" path="/var/lib/kubelet/pods/07faef23-22cf-4000-8966-eef38dbd15a8/volumes"
Sep 10 14:47:07.778228 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:07.778191    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"]
Sep 10 14:47:22.798100 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:22.798061    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc" podUID="f3419a59-e23a-4f3d-8f74-5ad1f5104eb6" containerName="main" containerID="cri-o://585e1865f3d97af24699f87b73b2bc75b5ccb14b5c366786b708fbae07e698d8" gracePeriod=45
Sep 10 14:47:23.051081 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.051027    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:47:23.153112 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.153083    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-model-cache\") pod \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") "
Sep 10 14:47:23.153304 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.153122    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-tls-certs\") pod \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") "
Sep 10 14:47:23.153304 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.153142    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-dshm\") pod \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") "
Sep 10 14:47:23.153304 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.153162    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-home\") pod \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") "
Sep 10 14:47:23.153304 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.153214    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqvlt\" (UniqueName: \"kubernetes.io/projected/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-kube-api-access-pqvlt\") pod \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") "
Sep 10 14:47:23.153304 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.153241    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-tmp-dir\") pod \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\" (UID: \"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6\") "
Sep 10 14:47:23.153576 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.153408    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3419a59-e23a-4f3d-8f74-5ad1f5104eb6" (UID: "f3419a59-e23a-4f3d-8f74-5ad1f5104eb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:23.153576 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.153486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-home" (OuterVolumeSpecName: "home") pod "f3419a59-e23a-4f3d-8f74-5ad1f5104eb6" (UID: "f3419a59-e23a-4f3d-8f74-5ad1f5104eb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:23.153576 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.153556    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3419a59-e23a-4f3d-8f74-5ad1f5104eb6" (UID: "f3419a59-e23a-4f3d-8f74-5ad1f5104eb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:23.153703 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.153600    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:47:23.155347 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.155322    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-dshm" (OuterVolumeSpecName: "dshm") pod "f3419a59-e23a-4f3d-8f74-5ad1f5104eb6" (UID: "f3419a59-e23a-4f3d-8f74-5ad1f5104eb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:23.155457 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.155396    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-kube-api-access-pqvlt" (OuterVolumeSpecName: "kube-api-access-pqvlt") pod "f3419a59-e23a-4f3d-8f74-5ad1f5104eb6" (UID: "f3419a59-e23a-4f3d-8f74-5ad1f5104eb6"). InnerVolumeSpecName "kube-api-access-pqvlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:47:23.155457 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.155424    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3419a59-e23a-4f3d-8f74-5ad1f5104eb6" (UID: "f3419a59-e23a-4f3d-8f74-5ad1f5104eb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:47:23.254279 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.254244    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:47:23.254279 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.254275    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqvlt\" (UniqueName: \"kubernetes.io/projected/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-kube-api-access-pqvlt\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:47:23.254279 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.254287    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:47:23.254532 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.254296    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:47:23.254532 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.254304    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:47:23.568164 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.568128    2580 generic.go:356] "Generic (PLEG): container finished" podID="f3419a59-e23a-4f3d-8f74-5ad1f5104eb6" containerID="585e1865f3d97af24699f87b73b2bc75b5ccb14b5c366786b708fbae07e698d8" exitCode=0
Sep 10 14:47:23.568336 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.568176    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"
Sep 10 14:47:23.568336 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.568189    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc" event={"ID":"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6","Type":"ContainerDied","Data":"585e1865f3d97af24699f87b73b2bc75b5ccb14b5c366786b708fbae07e698d8"}
Sep 10 14:47:23.568336 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.568226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc" event={"ID":"f3419a59-e23a-4f3d-8f74-5ad1f5104eb6","Type":"ContainerDied","Data":"17b17ce2ae6dd193c7ec98ade844a39c4f4419e690bc19874ebaa87fcab1b8ef"}
Sep 10 14:47:23.568336 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.568248    2580 scope.go:117] "RemoveContainer" containerID="585e1865f3d97af24699f87b73b2bc75b5ccb14b5c366786b708fbae07e698d8"
Sep 10 14:47:23.579764 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.579747    2580 scope.go:117] "RemoveContainer" containerID="585e1865f3d97af24699f87b73b2bc75b5ccb14b5c366786b708fbae07e698d8"
Sep 10 14:47:23.580081 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:47:23.580051    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585e1865f3d97af24699f87b73b2bc75b5ccb14b5c366786b708fbae07e698d8\": container with ID starting with 585e1865f3d97af24699f87b73b2bc75b5ccb14b5c366786b708fbae07e698d8 not found: ID does not exist" containerID="585e1865f3d97af24699f87b73b2bc75b5ccb14b5c366786b708fbae07e698d8"
Sep 10 14:47:23.580188 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.580092    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585e1865f3d97af24699f87b73b2bc75b5ccb14b5c366786b708fbae07e698d8"} err="failed to get container status \"585e1865f3d97af24699f87b73b2bc75b5ccb14b5c366786b708fbae07e698d8\": rpc error: code = NotFound desc = could not find container \"585e1865f3d97af24699f87b73b2bc75b5ccb14b5c366786b708fbae07e698d8\": container with ID starting with 585e1865f3d97af24699f87b73b2bc75b5ccb14b5c366786b708fbae07e698d8 not found: ID does not exist"
Sep 10 14:47:23.591295 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.591267    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"]
Sep 10 14:47:23.593469 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:23.593450    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-42514d/ipp-route-42514d-kserve-5ff45cd4cf-9gbmc"]
Sep 10 14:47:24.309817 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:47:24.309786    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3419a59-e23a-4f3d-8f74-5ad1f5104eb6" path="/var/lib/kubelet/pods/f3419a59-e23a-4f3d-8f74-5ad1f5104eb6/volumes"
Sep 10 14:48:01.721086 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:01.721033    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"]
Sep 10 14:48:01.721609 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:01.721323    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr" podUID="0b32f788-cd11-4988-b457-eff711e71fef" containerName="maas-api" containerID="cri-o://dd6939e164c7a3e0bb279d22a877726511f2ce9b2046bdea6b8c6ba094cc8e27" gracePeriod=30
Sep 10 14:48:01.975913 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:01.975840    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:48:02.105141 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.105104    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b32f788-cd11-4988-b457-eff711e71fef-metrics-certs\") pod \"0b32f788-cd11-4988-b457-eff711e71fef\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") "
Sep 10 14:48:02.105374 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.105170    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b32f788-cd11-4988-b457-eff711e71fef-maas-api-tls\") pod \"0b32f788-cd11-4988-b457-eff711e71fef\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") "
Sep 10 14:48:02.105374 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.105223    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zb4g\" (UniqueName: \"kubernetes.io/projected/0b32f788-cd11-4988-b457-eff711e71fef-kube-api-access-2zb4g\") pod \"0b32f788-cd11-4988-b457-eff711e71fef\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") "
Sep 10 14:48:02.105374 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.105263    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b32f788-cd11-4988-b457-eff711e71fef-openshift-service-ca\") pod \"0b32f788-cd11-4988-b457-eff711e71fef\" (UID: \"0b32f788-cd11-4988-b457-eff711e71fef\") "
Sep 10 14:48:02.105955 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.105921    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b32f788-cd11-4988-b457-eff711e71fef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b32f788-cd11-4988-b457-eff711e71fef" (UID: "0b32f788-cd11-4988-b457-eff711e71fef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:48:02.107613 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.107578    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b32f788-cd11-4988-b457-eff711e71fef-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0b32f788-cd11-4988-b457-eff711e71fef" (UID: "0b32f788-cd11-4988-b457-eff711e71fef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:48:02.107931 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.107914    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b32f788-cd11-4988-b457-eff711e71fef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b32f788-cd11-4988-b457-eff711e71fef" (UID: "0b32f788-cd11-4988-b457-eff711e71fef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:48:02.108072 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.108048    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b32f788-cd11-4988-b457-eff711e71fef-kube-api-access-2zb4g" (OuterVolumeSpecName: "kube-api-access-2zb4g") pod "0b32f788-cd11-4988-b457-eff711e71fef" (UID: "0b32f788-cd11-4988-b457-eff711e71fef"). InnerVolumeSpecName "kube-api-access-2zb4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:48:02.206587 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.206555    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b32f788-cd11-4988-b457-eff711e71fef-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:48:02.206587 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.206582    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b32f788-cd11-4988-b457-eff711e71fef-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:48:02.206587 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.206592    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zb4g\" (UniqueName: \"kubernetes.io/projected/0b32f788-cd11-4988-b457-eff711e71fef-kube-api-access-2zb4g\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:48:02.206830 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.206603    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b32f788-cd11-4988-b457-eff711e71fef-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:48:02.801088 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.801057    2580 generic.go:356] "Generic (PLEG): container finished" podID="0b32f788-cd11-4988-b457-eff711e71fef" containerID="dd6939e164c7a3e0bb279d22a877726511f2ce9b2046bdea6b8c6ba094cc8e27" exitCode=0
Sep 10 14:48:02.801518 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.801109    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"
Sep 10 14:48:02.801518 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.801140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr" event={"ID":"0b32f788-cd11-4988-b457-eff711e71fef","Type":"ContainerDied","Data":"dd6939e164c7a3e0bb279d22a877726511f2ce9b2046bdea6b8c6ba094cc8e27"}
Sep 10 14:48:02.801518 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.801194    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr" event={"ID":"0b32f788-cd11-4988-b457-eff711e71fef","Type":"ContainerDied","Data":"2603e7f5f6b84b28bd7967062ad3914448a70bd45fb2265bb5a628f5565d21d6"}
Sep 10 14:48:02.801518 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.801222    2580 scope.go:117] "RemoveContainer" containerID="dd6939e164c7a3e0bb279d22a877726511f2ce9b2046bdea6b8c6ba094cc8e27"
Sep 10 14:48:02.810195 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.810176    2580 scope.go:117] "RemoveContainer" containerID="dd6939e164c7a3e0bb279d22a877726511f2ce9b2046bdea6b8c6ba094cc8e27"
Sep 10 14:48:02.810460 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:48:02.810439    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6939e164c7a3e0bb279d22a877726511f2ce9b2046bdea6b8c6ba094cc8e27\": container with ID starting with dd6939e164c7a3e0bb279d22a877726511f2ce9b2046bdea6b8c6ba094cc8e27 not found: ID does not exist" containerID="dd6939e164c7a3e0bb279d22a877726511f2ce9b2046bdea6b8c6ba094cc8e27"
Sep 10 14:48:02.810500 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.810471    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6939e164c7a3e0bb279d22a877726511f2ce9b2046bdea6b8c6ba094cc8e27"} err="failed to get container status \"dd6939e164c7a3e0bb279d22a877726511f2ce9b2046bdea6b8c6ba094cc8e27\": rpc error: code = NotFound desc = could not find container \"dd6939e164c7a3e0bb279d22a877726511f2ce9b2046bdea6b8c6ba094cc8e27\": container with ID starting with dd6939e164c7a3e0bb279d22a877726511f2ce9b2046bdea6b8c6ba094cc8e27 not found: ID does not exist"
Sep 10 14:48:02.819187 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.819154    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"]
Sep 10 14:48:02.822954 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:02.822926    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c43af-55bddcb478-ldrtr"]
Sep 10 14:48:04.310634 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:04.310560    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b32f788-cd11-4988-b457-eff711e71fef" path="/var/lib/kubelet/pods/0b32f788-cd11-4988-b457-eff711e71fef/volumes"
Sep 10 14:48:19.316884 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.316850    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"]
Sep 10 14:48:19.317317 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.317079    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f" podUID="0865ec46-f028-4d9f-a97d-a60e37a487e8" containerName="maas-api" containerID="cri-o://b3086d32cf79e78e18852ffcaec1e07514878e79e36e53036f4ef6dddd9b4573" gracePeriod=30
Sep 10 14:48:19.556496 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.556466    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:48:19.662927 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.662899    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0865ec46-f028-4d9f-a97d-a60e37a487e8-maas-api-tls\") pod \"0865ec46-f028-4d9f-a97d-a60e37a487e8\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") "
Sep 10 14:48:19.663100 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.662968    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0865ec46-f028-4d9f-a97d-a60e37a487e8-metrics-certs\") pod \"0865ec46-f028-4d9f-a97d-a60e37a487e8\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") "
Sep 10 14:48:19.663151 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.663102    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtzpp\" (UniqueName: \"kubernetes.io/projected/0865ec46-f028-4d9f-a97d-a60e37a487e8-kube-api-access-dtzpp\") pod \"0865ec46-f028-4d9f-a97d-a60e37a487e8\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") "
Sep 10 14:48:19.663194 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.663165    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0865ec46-f028-4d9f-a97d-a60e37a487e8-openshift-service-ca\") pod \"0865ec46-f028-4d9f-a97d-a60e37a487e8\" (UID: \"0865ec46-f028-4d9f-a97d-a60e37a487e8\") "
Sep 10 14:48:19.663560 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.663526    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0865ec46-f028-4d9f-a97d-a60e37a487e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0865ec46-f028-4d9f-a97d-a60e37a487e8" (UID: "0865ec46-f028-4d9f-a97d-a60e37a487e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:48:19.665224 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.665189    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0865ec46-f028-4d9f-a97d-a60e37a487e8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0865ec46-f028-4d9f-a97d-a60e37a487e8" (UID: "0865ec46-f028-4d9f-a97d-a60e37a487e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:48:19.665224 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.665208    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0865ec46-f028-4d9f-a97d-a60e37a487e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0865ec46-f028-4d9f-a97d-a60e37a487e8" (UID: "0865ec46-f028-4d9f-a97d-a60e37a487e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:48:19.665376 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.665276    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0865ec46-f028-4d9f-a97d-a60e37a487e8-kube-api-access-dtzpp" (OuterVolumeSpecName: "kube-api-access-dtzpp") pod "0865ec46-f028-4d9f-a97d-a60e37a487e8" (UID: "0865ec46-f028-4d9f-a97d-a60e37a487e8"). InnerVolumeSpecName "kube-api-access-dtzpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:48:19.764661 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.764617    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0865ec46-f028-4d9f-a97d-a60e37a487e8-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:48:19.764661 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.764650    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0865ec46-f028-4d9f-a97d-a60e37a487e8-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:48:19.764661 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.764660    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0865ec46-f028-4d9f-a97d-a60e37a487e8-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:48:19.764661 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.764668    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtzpp\" (UniqueName: \"kubernetes.io/projected/0865ec46-f028-4d9f-a97d-a60e37a487e8-kube-api-access-dtzpp\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:48:19.896105 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.896070    2580 generic.go:356] "Generic (PLEG): container finished" podID="0865ec46-f028-4d9f-a97d-a60e37a487e8" containerID="b3086d32cf79e78e18852ffcaec1e07514878e79e36e53036f4ef6dddd9b4573" exitCode=0
Sep 10 14:48:19.896299 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.896113    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"
Sep 10 14:48:19.896299 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.896114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f" event={"ID":"0865ec46-f028-4d9f-a97d-a60e37a487e8","Type":"ContainerDied","Data":"b3086d32cf79e78e18852ffcaec1e07514878e79e36e53036f4ef6dddd9b4573"}
Sep 10 14:48:19.896299 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.896242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f" event={"ID":"0865ec46-f028-4d9f-a97d-a60e37a487e8","Type":"ContainerDied","Data":"c5c6e10ba829127c6a76415d51fc250f36c86f222de7bfff9f15851318315e97"}
Sep 10 14:48:19.896299 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.896273    2580 scope.go:117] "RemoveContainer" containerID="b3086d32cf79e78e18852ffcaec1e07514878e79e36e53036f4ef6dddd9b4573"
Sep 10 14:48:19.906265 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.906245    2580 scope.go:117] "RemoveContainer" containerID="b3086d32cf79e78e18852ffcaec1e07514878e79e36e53036f4ef6dddd9b4573"
Sep 10 14:48:19.906561 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:48:19.906539    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3086d32cf79e78e18852ffcaec1e07514878e79e36e53036f4ef6dddd9b4573\": container with ID starting with b3086d32cf79e78e18852ffcaec1e07514878e79e36e53036f4ef6dddd9b4573 not found: ID does not exist" containerID="b3086d32cf79e78e18852ffcaec1e07514878e79e36e53036f4ef6dddd9b4573"
Sep 10 14:48:19.906609 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.906571    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3086d32cf79e78e18852ffcaec1e07514878e79e36e53036f4ef6dddd9b4573"} err="failed to get container status \"b3086d32cf79e78e18852ffcaec1e07514878e79e36e53036f4ef6dddd9b4573\": rpc error: code = NotFound desc = could not find container \"b3086d32cf79e78e18852ffcaec1e07514878e79e36e53036f4ef6dddd9b4573\": container with ID starting with b3086d32cf79e78e18852ffcaec1e07514878e79e36e53036f4ef6dddd9b4573 not found: ID does not exist"
Sep 10 14:48:19.920115 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.920035    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"]
Sep 10 14:48:19.922349 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:19.922324    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7bf16f-5f96dff7c6-h5n9f"]
Sep 10 14:48:20.310069 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:20.309966    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0865ec46-f028-4d9f-a97d-a60e37a487e8" path="/var/lib/kubelet/pods/0865ec46-f028-4d9f-a97d-a60e37a487e8/volumes"
Sep 10 14:48:36.890997 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:36.890947    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"]
Sep 10 14:48:36.891461 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:36.891227    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h" podUID="85029c90-a894-4382-8620-4847ad3d1409" containerName="maas-api" containerID="cri-o://9fdb31465fe12483a54b38db07c6b33f76e1885604cb67b904897dd6a274df1a" gracePeriod=30
Sep 10 14:48:37.146769 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.146705    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:48:37.242073 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.242041    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85029c90-a894-4382-8620-4847ad3d1409-openshift-service-ca\") pod \"85029c90-a894-4382-8620-4847ad3d1409\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") "
Sep 10 14:48:37.242227 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.242080    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85029c90-a894-4382-8620-4847ad3d1409-maas-api-tls\") pod \"85029c90-a894-4382-8620-4847ad3d1409\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") "
Sep 10 14:48:37.242227 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.242106    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85029c90-a894-4382-8620-4847ad3d1409-metrics-certs\") pod \"85029c90-a894-4382-8620-4847ad3d1409\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") "
Sep 10 14:48:37.242227 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.242170    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm9gr\" (UniqueName: \"kubernetes.io/projected/85029c90-a894-4382-8620-4847ad3d1409-kube-api-access-mm9gr\") pod \"85029c90-a894-4382-8620-4847ad3d1409\" (UID: \"85029c90-a894-4382-8620-4847ad3d1409\") "
Sep 10 14:48:37.242486 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.242455    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85029c90-a894-4382-8620-4847ad3d1409-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "85029c90-a894-4382-8620-4847ad3d1409" (UID: "85029c90-a894-4382-8620-4847ad3d1409"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:48:37.244305 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.244274    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85029c90-a894-4382-8620-4847ad3d1409-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "85029c90-a894-4382-8620-4847ad3d1409" (UID: "85029c90-a894-4382-8620-4847ad3d1409"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:48:37.244305 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.244297    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85029c90-a894-4382-8620-4847ad3d1409-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "85029c90-a894-4382-8620-4847ad3d1409" (UID: "85029c90-a894-4382-8620-4847ad3d1409"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:48:37.244470 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.244441    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85029c90-a894-4382-8620-4847ad3d1409-kube-api-access-mm9gr" (OuterVolumeSpecName: "kube-api-access-mm9gr") pod "85029c90-a894-4382-8620-4847ad3d1409" (UID: "85029c90-a894-4382-8620-4847ad3d1409"). InnerVolumeSpecName "kube-api-access-mm9gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:48:37.343783 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.343742    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm9gr\" (UniqueName: \"kubernetes.io/projected/85029c90-a894-4382-8620-4847ad3d1409-kube-api-access-mm9gr\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:48:37.343783 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.343778    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85029c90-a894-4382-8620-4847ad3d1409-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:48:37.344020 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.343793    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85029c90-a894-4382-8620-4847ad3d1409-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:48:37.344020 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.343807    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85029c90-a894-4382-8620-4847ad3d1409-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:48:37.996386 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.996345    2580 generic.go:356] "Generic (PLEG): container finished" podID="85029c90-a894-4382-8620-4847ad3d1409" containerID="9fdb31465fe12483a54b38db07c6b33f76e1885604cb67b904897dd6a274df1a" exitCode=0
Sep 10 14:48:37.996854 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.996395    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"
Sep 10 14:48:37.996854 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.996438    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h" event={"ID":"85029c90-a894-4382-8620-4847ad3d1409","Type":"ContainerDied","Data":"9fdb31465fe12483a54b38db07c6b33f76e1885604cb67b904897dd6a274df1a"}
Sep 10 14:48:37.996854 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.996506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h" event={"ID":"85029c90-a894-4382-8620-4847ad3d1409","Type":"ContainerDied","Data":"b01713405aefd5859e863ea5082d5aed493529fd8bb9ada7a75c0a86efb51a0e"}
Sep 10 14:48:37.996854 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:37.996541    2580 scope.go:117] "RemoveContainer" containerID="9fdb31465fe12483a54b38db07c6b33f76e1885604cb67b904897dd6a274df1a"
Sep 10 14:48:38.006078 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:38.006058    2580 scope.go:117] "RemoveContainer" containerID="9fdb31465fe12483a54b38db07c6b33f76e1885604cb67b904897dd6a274df1a"
Sep 10 14:48:38.006390 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:48:38.006372    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fdb31465fe12483a54b38db07c6b33f76e1885604cb67b904897dd6a274df1a\": container with ID starting with 9fdb31465fe12483a54b38db07c6b33f76e1885604cb67b904897dd6a274df1a not found: ID does not exist" containerID="9fdb31465fe12483a54b38db07c6b33f76e1885604cb67b904897dd6a274df1a"
Sep 10 14:48:38.006430 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:38.006400    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fdb31465fe12483a54b38db07c6b33f76e1885604cb67b904897dd6a274df1a"} err="failed to get container status \"9fdb31465fe12483a54b38db07c6b33f76e1885604cb67b904897dd6a274df1a\": rpc error: code = NotFound desc = could not find container \"9fdb31465fe12483a54b38db07c6b33f76e1885604cb67b904897dd6a274df1a\": container with ID starting with 9fdb31465fe12483a54b38db07c6b33f76e1885604cb67b904897dd6a274df1a not found: ID does not exist"
Sep 10 14:48:38.019536 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:38.019506    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"]
Sep 10 14:48:38.023574 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:38.023549    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-59dc59-8586b49676-m6d4h"]
Sep 10 14:48:38.310097 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:38.310021    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85029c90-a894-4382-8620-4847ad3d1409" path="/var/lib/kubelet/pods/85029c90-a894-4382-8620-4847ad3d1409/volumes"
Sep 10 14:48:42.643949 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:42.643914    2580 scope.go:117] "RemoveContainer" containerID="e32025361338ef9a3f7d60b38d8dca5910a3eef237487b7dfa3132c04f488efc"
Sep 10 14:48:46.323582 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:46.323544    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6857cb7d9d-gst9r"]
Sep 10 14:48:46.324021 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:46.323777    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6857cb7d9d-gst9r" podUID="7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d" containerName="manager" containerID="cri-o://fff57f019b5dd1583a3d32a3e62ecc3499233f39246ff6c2fbf0bd85004a9498" gracePeriod=30
Sep 10 14:48:46.572540 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:46.572517    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:48:46.627184 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:46.627100    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgzz8\" (UniqueName: \"kubernetes.io/projected/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-kube-api-access-tgzz8\") pod \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\" (UID: \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\") "
Sep 10 14:48:46.627184 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:46.627174    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-metrics-certs\") pod \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\" (UID: \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\") "
Sep 10 14:48:46.627428 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:46.627216    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-webhook-cert\") pod \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\" (UID: \"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d\") "
Sep 10 14:48:46.629226 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:46.629190    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d" (UID: "7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:48:46.629347 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:46.629296    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-kube-api-access-tgzz8" (OuterVolumeSpecName: "kube-api-access-tgzz8") pod "7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d" (UID: "7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d"). InnerVolumeSpecName "kube-api-access-tgzz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:48:46.629347 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:46.629306    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d" (UID: "7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:48:46.728260 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:46.728220    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgzz8\" (UniqueName: \"kubernetes.io/projected/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-kube-api-access-tgzz8\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:48:46.728260 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:46.728251    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:48:46.728260 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:46.728261    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d-webhook-cert\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:48:47.057225 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:47.050183    2580 generic.go:356] "Generic (PLEG): container finished" podID="7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d" containerID="fff57f019b5dd1583a3d32a3e62ecc3499233f39246ff6c2fbf0bd85004a9498" exitCode=0
Sep 10 14:48:47.057225 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:47.050459    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6857cb7d9d-gst9r" event={"ID":"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d","Type":"ContainerDied","Data":"fff57f019b5dd1583a3d32a3e62ecc3499233f39246ff6c2fbf0bd85004a9498"}
Sep 10 14:48:47.057225 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:47.050597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6857cb7d9d-gst9r" event={"ID":"7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d","Type":"ContainerDied","Data":"fc05810e88f0bab0c189a4fa923cf2f40945d9421392a43222f8bb2e548fd0a4"}
Sep 10 14:48:47.057225 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:47.050653    2580 scope.go:117] "RemoveContainer" containerID="fff57f019b5dd1583a3d32a3e62ecc3499233f39246ff6c2fbf0bd85004a9498"
Sep 10 14:48:47.057225 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:47.050810    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6857cb7d9d-gst9r"
Sep 10 14:48:47.066914 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:47.066892    2580 scope.go:117] "RemoveContainer" containerID="fff57f019b5dd1583a3d32a3e62ecc3499233f39246ff6c2fbf0bd85004a9498"
Sep 10 14:48:47.067238 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:48:47.067216    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff57f019b5dd1583a3d32a3e62ecc3499233f39246ff6c2fbf0bd85004a9498\": container with ID starting with fff57f019b5dd1583a3d32a3e62ecc3499233f39246ff6c2fbf0bd85004a9498 not found: ID does not exist" containerID="fff57f019b5dd1583a3d32a3e62ecc3499233f39246ff6c2fbf0bd85004a9498"
Sep 10 14:48:47.067310 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:47.067245    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff57f019b5dd1583a3d32a3e62ecc3499233f39246ff6c2fbf0bd85004a9498"} err="failed to get container status \"fff57f019b5dd1583a3d32a3e62ecc3499233f39246ff6c2fbf0bd85004a9498\": rpc error: code = NotFound desc = could not find container \"fff57f019b5dd1583a3d32a3e62ecc3499233f39246ff6c2fbf0bd85004a9498\": container with ID starting with fff57f019b5dd1583a3d32a3e62ecc3499233f39246ff6c2fbf0bd85004a9498 not found: ID does not exist"
Sep 10 14:48:47.076515 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:47.076492    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6857cb7d9d-gst9r"]
Sep 10 14:48:47.079586 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:47.079561    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6857cb7d9d-gst9r"]
Sep 10 14:48:48.310187 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:48:48.310158    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d" path="/var/lib/kubelet/pods/7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d/volumes"
Sep 10 14:49:42.419794 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:42.419766    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:49:42.431924 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:42.431897    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:49:47.877316 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877275    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6857cb7d9d-5m972"]
Sep 10 14:49:47.877697 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877650    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3419a59-e23a-4f3d-8f74-5ad1f5104eb6" containerName="main"
Sep 10 14:49:47.877697 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877662    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3419a59-e23a-4f3d-8f74-5ad1f5104eb6" containerName="main"
Sep 10 14:49:47.877697 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877672    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b32f788-cd11-4988-b457-eff711e71fef" containerName="maas-api"
Sep 10 14:49:47.877697 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877680    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b32f788-cd11-4988-b457-eff711e71fef" containerName="maas-api"
Sep 10 14:49:47.877697 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877697    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5074be9b-503b-47dd-a59e-0e6632cb30e4" containerName="maas-api"
Sep 10 14:49:47.877862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877704    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5074be9b-503b-47dd-a59e-0e6632cb30e4" containerName="maas-api"
Sep 10 14:49:47.877862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877710    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0865ec46-f028-4d9f-a97d-a60e37a487e8" containerName="maas-api"
Sep 10 14:49:47.877862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877715    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0865ec46-f028-4d9f-a97d-a60e37a487e8" containerName="maas-api"
Sep 10 14:49:47.877862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877724    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07faef23-22cf-4000-8966-eef38dbd15a8" containerName="maas-api"
Sep 10 14:49:47.877862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877729    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07faef23-22cf-4000-8966-eef38dbd15a8" containerName="maas-api"
Sep 10 14:49:47.877862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877736    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d" containerName="manager"
Sep 10 14:49:47.877862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877741    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d" containerName="manager"
Sep 10 14:49:47.877862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877748    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85029c90-a894-4382-8620-4847ad3d1409" containerName="maas-api"
Sep 10 14:49:47.877862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877752    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85029c90-a894-4382-8620-4847ad3d1409" containerName="maas-api"
Sep 10 14:49:47.877862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877825    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07faef23-22cf-4000-8966-eef38dbd15a8" containerName="maas-api"
Sep 10 14:49:47.877862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877833    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b32f788-cd11-4988-b457-eff711e71fef" containerName="maas-api"
Sep 10 14:49:47.877862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877843    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0865ec46-f028-4d9f-a97d-a60e37a487e8" containerName="maas-api"
Sep 10 14:49:47.877862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877850    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="85029c90-a894-4382-8620-4847ad3d1409" containerName="maas-api"
Sep 10 14:49:47.877862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877858    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3419a59-e23a-4f3d-8f74-5ad1f5104eb6" containerName="main"
Sep 10 14:49:47.877862 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877864    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5074be9b-503b-47dd-a59e-0e6632cb30e4" containerName="maas-api"
Sep 10 14:49:47.878335 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.877870    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7af9765c-c2d0-4c1b-86a4-9aeb6cd62e5d" containerName="manager"
Sep 10 14:49:47.880704 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.880688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6857cb7d9d-5m972"
Sep 10 14:49:47.884215 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.884191    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 14:49:47.884354 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.884226    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 14:49:47.884354 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.884295    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tbsjp\""
Sep 10 14:49:47.884354 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.884226    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 14:49:47.890613 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.890580    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6857cb7d9d-5m972"]
Sep 10 14:49:47.954821 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.954779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb2kk\" (UniqueName: \"kubernetes.io/projected/cde6579f-a43c-4789-9ebb-cbc79b165e5b-kube-api-access-nb2kk\") pod \"maas-controller-6857cb7d9d-5m972\" (UID: \"cde6579f-a43c-4789-9ebb-cbc79b165e5b\") " pod="opendatahub/maas-controller-6857cb7d9d-5m972"
Sep 10 14:49:47.954821 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.954826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde6579f-a43c-4789-9ebb-cbc79b165e5b-metrics-certs\") pod \"maas-controller-6857cb7d9d-5m972\" (UID: \"cde6579f-a43c-4789-9ebb-cbc79b165e5b\") " pod="opendatahub/maas-controller-6857cb7d9d-5m972"
Sep 10 14:49:47.955101 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:47.954902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cde6579f-a43c-4789-9ebb-cbc79b165e5b-webhook-cert\") pod \"maas-controller-6857cb7d9d-5m972\" (UID: \"cde6579f-a43c-4789-9ebb-cbc79b165e5b\") " pod="opendatahub/maas-controller-6857cb7d9d-5m972"
Sep 10 14:49:48.056179 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:48.056142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb2kk\" (UniqueName: \"kubernetes.io/projected/cde6579f-a43c-4789-9ebb-cbc79b165e5b-kube-api-access-nb2kk\") pod \"maas-controller-6857cb7d9d-5m972\" (UID: \"cde6579f-a43c-4789-9ebb-cbc79b165e5b\") " pod="opendatahub/maas-controller-6857cb7d9d-5m972"
Sep 10 14:49:48.056307 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:48.056186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde6579f-a43c-4789-9ebb-cbc79b165e5b-metrics-certs\") pod \"maas-controller-6857cb7d9d-5m972\" (UID: \"cde6579f-a43c-4789-9ebb-cbc79b165e5b\") " pod="opendatahub/maas-controller-6857cb7d9d-5m972"
Sep 10 14:49:48.056307 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:48.056249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cde6579f-a43c-4789-9ebb-cbc79b165e5b-webhook-cert\") pod \"maas-controller-6857cb7d9d-5m972\" (UID: \"cde6579f-a43c-4789-9ebb-cbc79b165e5b\") " pod="opendatahub/maas-controller-6857cb7d9d-5m972"
Sep 10 14:49:48.058708 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:48.058681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cde6579f-a43c-4789-9ebb-cbc79b165e5b-webhook-cert\") pod \"maas-controller-6857cb7d9d-5m972\" (UID: \"cde6579f-a43c-4789-9ebb-cbc79b165e5b\") " pod="opendatahub/maas-controller-6857cb7d9d-5m972"
Sep 10 14:49:48.058708 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:48.058701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde6579f-a43c-4789-9ebb-cbc79b165e5b-metrics-certs\") pod \"maas-controller-6857cb7d9d-5m972\" (UID: \"cde6579f-a43c-4789-9ebb-cbc79b165e5b\") " pod="opendatahub/maas-controller-6857cb7d9d-5m972"
Sep 10 14:49:48.065822 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:48.065801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb2kk\" (UniqueName: \"kubernetes.io/projected/cde6579f-a43c-4789-9ebb-cbc79b165e5b-kube-api-access-nb2kk\") pod \"maas-controller-6857cb7d9d-5m972\" (UID: \"cde6579f-a43c-4789-9ebb-cbc79b165e5b\") " pod="opendatahub/maas-controller-6857cb7d9d-5m972"
Sep 10 14:49:48.191614 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:48.191580    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6857cb7d9d-5m972"
Sep 10 14:49:48.464848 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:48.464818    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6857cb7d9d-5m972"]
Sep 10 14:49:48.466140 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:49:48.466111    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde6579f_a43c_4789_9ebb_cbc79b165e5b.slice/crio-18e58f54545ce3df790ddb8720eebfd0152c9c195f388d059028376461b3e22c WatchSource:0}: Error finding container 18e58f54545ce3df790ddb8720eebfd0152c9c195f388d059028376461b3e22c: Status 404 returned error can't find the container with id 18e58f54545ce3df790ddb8720eebfd0152c9c195f388d059028376461b3e22c
Sep 10 14:49:48.467514 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:48.467496    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:49:49.387639 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:49.387606    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6857cb7d9d-5m972" event={"ID":"cde6579f-a43c-4789-9ebb-cbc79b165e5b","Type":"ContainerStarted","Data":"b836dc5b1ea5813468b3f3fca89463e0988c7fef411315f762c2837edce74db4"}
Sep 10 14:49:49.388077 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:49.387669    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6857cb7d9d-5m972"
Sep 10 14:49:49.388077 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:49.387694    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6857cb7d9d-5m972" event={"ID":"cde6579f-a43c-4789-9ebb-cbc79b165e5b","Type":"ContainerStarted","Data":"18e58f54545ce3df790ddb8720eebfd0152c9c195f388d059028376461b3e22c"}
Sep 10 14:49:49.406724 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:49:49.406657    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6857cb7d9d-5m972" podStartSLOduration=1.764796258 podStartE2EDuration="2.40663851s" totalImagesPullingTime="641.842252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:49.40627139 +0000 UTC m=+1807.597799612" watchObservedRunningTime="2026-09-10 14:49:49.40663851 +0000 UTC m=+1807.598166711"
Sep 10 14:50:00.395728 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:00.395696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6857cb7d9d-5m972"
Sep 10 14:50:11.823843 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:11.823805    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6857cb7d9d-5m972"]
Sep 10 14:50:11.824333 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:11.824043    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6857cb7d9d-5m972" podUID="cde6579f-a43c-4789-9ebb-cbc79b165e5b" containerName="manager" containerID="cri-o://b836dc5b1ea5813468b3f3fca89463e0988c7fef411315f762c2837edce74db4" gracePeriod=30
Sep 10 14:50:12.164747 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.164720    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6857cb7d9d-5m972"
Sep 10 14:50:12.275924 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.275891    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde6579f-a43c-4789-9ebb-cbc79b165e5b-metrics-certs\") pod \"cde6579f-a43c-4789-9ebb-cbc79b165e5b\" (UID: \"cde6579f-a43c-4789-9ebb-cbc79b165e5b\") "
Sep 10 14:50:12.276103 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.276054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb2kk\" (UniqueName: \"kubernetes.io/projected/cde6579f-a43c-4789-9ebb-cbc79b165e5b-kube-api-access-nb2kk\") pod \"cde6579f-a43c-4789-9ebb-cbc79b165e5b\" (UID: \"cde6579f-a43c-4789-9ebb-cbc79b165e5b\") "
Sep 10 14:50:12.276103 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.276096    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cde6579f-a43c-4789-9ebb-cbc79b165e5b-webhook-cert\") pod \"cde6579f-a43c-4789-9ebb-cbc79b165e5b\" (UID: \"cde6579f-a43c-4789-9ebb-cbc79b165e5b\") "
Sep 10 14:50:12.278108 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.278081    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde6579f-a43c-4789-9ebb-cbc79b165e5b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cde6579f-a43c-4789-9ebb-cbc79b165e5b" (UID: "cde6579f-a43c-4789-9ebb-cbc79b165e5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:50:12.278247 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.278109    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde6579f-a43c-4789-9ebb-cbc79b165e5b-kube-api-access-nb2kk" (OuterVolumeSpecName: "kube-api-access-nb2kk") pod "cde6579f-a43c-4789-9ebb-cbc79b165e5b" (UID: "cde6579f-a43c-4789-9ebb-cbc79b165e5b"). InnerVolumeSpecName "kube-api-access-nb2kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:50:12.278247 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.278170    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde6579f-a43c-4789-9ebb-cbc79b165e5b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "cde6579f-a43c-4789-9ebb-cbc79b165e5b" (UID: "cde6579f-a43c-4789-9ebb-cbc79b165e5b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:50:12.377750 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.377664    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb2kk\" (UniqueName: \"kubernetes.io/projected/cde6579f-a43c-4789-9ebb-cbc79b165e5b-kube-api-access-nb2kk\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:50:12.377750 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.377692    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cde6579f-a43c-4789-9ebb-cbc79b165e5b-webhook-cert\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:50:12.377750 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.377701    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde6579f-a43c-4789-9ebb-cbc79b165e5b-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:50:12.507308 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.507269    2580 generic.go:356] "Generic (PLEG): container finished" podID="cde6579f-a43c-4789-9ebb-cbc79b165e5b" containerID="b836dc5b1ea5813468b3f3fca89463e0988c7fef411315f762c2837edce74db4" exitCode=0
Sep 10 14:50:12.507506 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.507318    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6857cb7d9d-5m972"
Sep 10 14:50:12.507506 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.507358    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6857cb7d9d-5m972" event={"ID":"cde6579f-a43c-4789-9ebb-cbc79b165e5b","Type":"ContainerDied","Data":"b836dc5b1ea5813468b3f3fca89463e0988c7fef411315f762c2837edce74db4"}
Sep 10 14:50:12.507506 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.507416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6857cb7d9d-5m972" event={"ID":"cde6579f-a43c-4789-9ebb-cbc79b165e5b","Type":"ContainerDied","Data":"18e58f54545ce3df790ddb8720eebfd0152c9c195f388d059028376461b3e22c"}
Sep 10 14:50:12.507506 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.507447    2580 scope.go:117] "RemoveContainer" containerID="b836dc5b1ea5813468b3f3fca89463e0988c7fef411315f762c2837edce74db4"
Sep 10 14:50:12.516532 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.516514    2580 scope.go:117] "RemoveContainer" containerID="b836dc5b1ea5813468b3f3fca89463e0988c7fef411315f762c2837edce74db4"
Sep 10 14:50:12.516823 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:50:12.516806    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b836dc5b1ea5813468b3f3fca89463e0988c7fef411315f762c2837edce74db4\": container with ID starting with b836dc5b1ea5813468b3f3fca89463e0988c7fef411315f762c2837edce74db4 not found: ID does not exist" containerID="b836dc5b1ea5813468b3f3fca89463e0988c7fef411315f762c2837edce74db4"
Sep 10 14:50:12.516900 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.516833    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b836dc5b1ea5813468b3f3fca89463e0988c7fef411315f762c2837edce74db4"} err="failed to get container status \"b836dc5b1ea5813468b3f3fca89463e0988c7fef411315f762c2837edce74db4\": rpc error: code = NotFound desc = could not find container \"b836dc5b1ea5813468b3f3fca89463e0988c7fef411315f762c2837edce74db4\": container with ID starting with b836dc5b1ea5813468b3f3fca89463e0988c7fef411315f762c2837edce74db4 not found: ID does not exist"
Sep 10 14:50:12.523833 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.523805    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6857cb7d9d-5m972"]
Sep 10 14:50:12.525957 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:12.525936    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6857cb7d9d-5m972"]
Sep 10 14:50:13.284856 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.284814    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6857cb7d9d-tmvsw"]
Sep 10 14:50:13.285380 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.285359    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde6579f-a43c-4789-9ebb-cbc79b165e5b" containerName="manager"
Sep 10 14:50:13.285461 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.285382    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde6579f-a43c-4789-9ebb-cbc79b165e5b" containerName="manager"
Sep 10 14:50:13.285542 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.285529    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde6579f-a43c-4789-9ebb-cbc79b165e5b" containerName="manager"
Sep 10 14:50:13.289845 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.289824    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6857cb7d9d-tmvsw"
Sep 10 14:50:13.293363 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.293338    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tbsjp\""
Sep 10 14:50:13.293470 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.293340    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 14:50:13.293470 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.293341    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 14:50:13.293470 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.293382    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 14:50:13.298418 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.298396    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6857cb7d9d-tmvsw"]
Sep 10 14:50:13.388743 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.388705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbdkp\" (UniqueName: \"kubernetes.io/projected/133328f1-4c9c-4289-b248-5e16dd22cdc0-kube-api-access-xbdkp\") pod \"maas-controller-6857cb7d9d-tmvsw\" (UID: \"133328f1-4c9c-4289-b248-5e16dd22cdc0\") " pod="opendatahub/maas-controller-6857cb7d9d-tmvsw"
Sep 10 14:50:13.388919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.388758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/133328f1-4c9c-4289-b248-5e16dd22cdc0-metrics-certs\") pod \"maas-controller-6857cb7d9d-tmvsw\" (UID: \"133328f1-4c9c-4289-b248-5e16dd22cdc0\") " pod="opendatahub/maas-controller-6857cb7d9d-tmvsw"
Sep 10 14:50:13.388919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.388865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/133328f1-4c9c-4289-b248-5e16dd22cdc0-webhook-cert\") pod \"maas-controller-6857cb7d9d-tmvsw\" (UID: \"133328f1-4c9c-4289-b248-5e16dd22cdc0\") " pod="opendatahub/maas-controller-6857cb7d9d-tmvsw"
Sep 10 14:50:13.489791 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.489746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/133328f1-4c9c-4289-b248-5e16dd22cdc0-webhook-cert\") pod \"maas-controller-6857cb7d9d-tmvsw\" (UID: \"133328f1-4c9c-4289-b248-5e16dd22cdc0\") " pod="opendatahub/maas-controller-6857cb7d9d-tmvsw"
Sep 10 14:50:13.489961 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.489823    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbdkp\" (UniqueName: \"kubernetes.io/projected/133328f1-4c9c-4289-b248-5e16dd22cdc0-kube-api-access-xbdkp\") pod \"maas-controller-6857cb7d9d-tmvsw\" (UID: \"133328f1-4c9c-4289-b248-5e16dd22cdc0\") " pod="opendatahub/maas-controller-6857cb7d9d-tmvsw"
Sep 10 14:50:13.489961 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.489867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/133328f1-4c9c-4289-b248-5e16dd22cdc0-metrics-certs\") pod \"maas-controller-6857cb7d9d-tmvsw\" (UID: \"133328f1-4c9c-4289-b248-5e16dd22cdc0\") " pod="opendatahub/maas-controller-6857cb7d9d-tmvsw"
Sep 10 14:50:13.492346 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.492326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/133328f1-4c9c-4289-b248-5e16dd22cdc0-metrics-certs\") pod \"maas-controller-6857cb7d9d-tmvsw\" (UID: \"133328f1-4c9c-4289-b248-5e16dd22cdc0\") " pod="opendatahub/maas-controller-6857cb7d9d-tmvsw"
Sep 10 14:50:13.492424 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.492387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/133328f1-4c9c-4289-b248-5e16dd22cdc0-webhook-cert\") pod \"maas-controller-6857cb7d9d-tmvsw\" (UID: \"133328f1-4c9c-4289-b248-5e16dd22cdc0\") " pod="opendatahub/maas-controller-6857cb7d9d-tmvsw"
Sep 10 14:50:13.506031 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.498911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbdkp\" (UniqueName: \"kubernetes.io/projected/133328f1-4c9c-4289-b248-5e16dd22cdc0-kube-api-access-xbdkp\") pod \"maas-controller-6857cb7d9d-tmvsw\" (UID: \"133328f1-4c9c-4289-b248-5e16dd22cdc0\") " pod="opendatahub/maas-controller-6857cb7d9d-tmvsw"
Sep 10 14:50:13.602006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.601888    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6857cb7d9d-tmvsw"
Sep 10 14:50:13.874941 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:13.874914    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6857cb7d9d-tmvsw"]
Sep 10 14:50:13.877167 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:50:13.877141    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133328f1_4c9c_4289_b248_5e16dd22cdc0.slice/crio-a470d59c70d741c40907b7e8f9db938a4b548d0d1161b1a23892ba4041bd36a1 WatchSource:0}: Error finding container a470d59c70d741c40907b7e8f9db938a4b548d0d1161b1a23892ba4041bd36a1: Status 404 returned error can't find the container with id a470d59c70d741c40907b7e8f9db938a4b548d0d1161b1a23892ba4041bd36a1
Sep 10 14:50:14.310903 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:14.310878    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde6579f-a43c-4789-9ebb-cbc79b165e5b" path="/var/lib/kubelet/pods/cde6579f-a43c-4789-9ebb-cbc79b165e5b/volumes"
Sep 10 14:50:14.523896 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:14.523861    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6857cb7d9d-tmvsw" event={"ID":"133328f1-4c9c-4289-b248-5e16dd22cdc0","Type":"ContainerStarted","Data":"d921222cc9be5995924e5549fe770c3d040aface05ab947b446001eae3340a5f"}
Sep 10 14:50:14.524089 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:14.523920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6857cb7d9d-tmvsw" event={"ID":"133328f1-4c9c-4289-b248-5e16dd22cdc0","Type":"ContainerStarted","Data":"a470d59c70d741c40907b7e8f9db938a4b548d0d1161b1a23892ba4041bd36a1"}
Sep 10 14:50:14.524089 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:14.524012    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6857cb7d9d-tmvsw"
Sep 10 14:50:14.543082 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:14.543030    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6857cb7d9d-tmvsw" podStartSLOduration=1.119409388 podStartE2EDuration="1.543011076s" totalImagesPullingTime="423.601688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:50:14.542377546 +0000 UTC m=+1832.733905745" watchObservedRunningTime="2026-09-10 14:50:14.543011076 +0000 UTC m=+1832.734539276"
Sep 10 14:50:25.532006 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:25.531958    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6857cb7d9d-tmvsw"
Sep 10 14:50:42.699708 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:50:42.699666    2580 scope.go:117] "RemoveContainer" containerID="f512bb8dbd7fb35350d4aad3d3291463822e970e7ee2edf063dd1129389a6c25"
Sep 10 14:53:24.916131 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:24.916095    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"]
Sep 10 14:53:24.918996 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:24.918960    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:24.921384 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:24.921360    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f681ae9c-kserve-self-signed-certs\""
Sep 10 14:53:24.927774 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:24.927746    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"]
Sep 10 14:53:24.967611 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:24.967577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549350ab-b09e-41bb-8162-1046d9c4c263-tls-certs\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:24.967792 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:24.967623    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-tmp-dir\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:24.967792 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:24.967646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-home\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:24.967792 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:24.967696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm6qv\" (UniqueName: \"kubernetes.io/projected/549350ab-b09e-41bb-8162-1046d9c4c263-kube-api-access-zm6qv\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:24.967792 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:24.967733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-dshm\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:24.967792 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:24.967766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-model-cache\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:25.069069 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:25.069029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-home\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:25.069069 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:25.069073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm6qv\" (UniqueName: \"kubernetes.io/projected/549350ab-b09e-41bb-8162-1046d9c4c263-kube-api-access-zm6qv\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:25.069344 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:25.069097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-dshm\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:25.069344 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:25.069121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-model-cache\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:25.069344 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:25.069204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549350ab-b09e-41bb-8162-1046d9c4c263-tls-certs\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:25.069344 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:25.069242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-tmp-dir\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:25.069623 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:25.069591    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-home\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:25.069623 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:25.069614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-tmp-dir\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:25.069722 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:25.069642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-model-cache\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:25.071307 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:25.071285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-dshm\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:25.071584 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:25.071565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549350ab-b09e-41bb-8162-1046d9c4c263-tls-certs\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:25.076421 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:25.076397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm6qv\" (UniqueName: \"kubernetes.io/projected/549350ab-b09e-41bb-8162-1046d9c4c263-kube-api-access-zm6qv\") pod \"e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") " pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:25.228425 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:25.228331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:25.500449 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:25.500419    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"]
Sep 10 14:53:25.502504 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:53:25.502473    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod549350ab_b09e_41bb_8162_1046d9c4c263.slice/crio-be4e019dada971b1a1cc58a3cda07a4eb557cb24aabbfca18c31abb7e3763699 WatchSource:0}: Error finding container be4e019dada971b1a1cc58a3cda07a4eb557cb24aabbfca18c31abb7e3763699: Status 404 returned error can't find the container with id be4e019dada971b1a1cc58a3cda07a4eb557cb24aabbfca18c31abb7e3763699
Sep 10 14:53:25.532094 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:25.532062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r" event={"ID":"549350ab-b09e-41bb-8162-1046d9c4c263","Type":"ContainerStarted","Data":"be4e019dada971b1a1cc58a3cda07a4eb557cb24aabbfca18c31abb7e3763699"}
Sep 10 14:53:26.544822 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:26.544781    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r" event={"ID":"549350ab-b09e-41bb-8162-1046d9c4c263","Type":"ContainerStarted","Data":"9e93548e109e4e04dde5129e6d2f5da78f4b7fd762809046f736e68c4f7e098a"}
Sep 10 14:53:26.561855 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:26.561803    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r" podStartSLOduration=2.561788407 podStartE2EDuration="2.561788407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:53:26.560600883 +0000 UTC m=+2024.752129076" watchObservedRunningTime="2026-09-10 14:53:26.561788407 +0000 UTC m=+2024.753316605"
Sep 10 14:53:35.228664 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:35.228620    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:35.228664 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:35.228676    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:35.241163 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:35.241140    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:35.602526 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:35.602438    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:53:46.214532 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.214498    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"]
Sep 10 14:53:46.218070 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.218045    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.220617 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.220594    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f681ae9c-kserve-self-signed-certs\""
Sep 10 14:53:46.227542 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.227516    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"]
Sep 10 14:53:46.270011 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.269955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-dshm\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.270192 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.270017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gvp\" (UniqueName: \"kubernetes.io/projected/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-kube-api-access-f4gvp\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.270192 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.270067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-tmp-dir\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.270192 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.270152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-tls-certs\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.270326 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.270197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-model-cache\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.270326 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.270242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-home\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.371049 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.371012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-tls-certs\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.371049 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.371057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-model-cache\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.371316 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.371088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-home\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.371316 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.371119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-dshm\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.371316 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.371137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gvp\" (UniqueName: \"kubernetes.io/projected/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-kube-api-access-f4gvp\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.371316 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.371205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-tmp-dir\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.371613 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.371579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-home\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.371690 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.371626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-tmp-dir\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.371690 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.371641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-model-cache\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.373348 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.373320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-dshm\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.373675 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.373653    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-tls-certs\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.379318 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.379296    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gvp\" (UniqueName: \"kubernetes.io/projected/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-kube-api-access-f4gvp\") pod \"e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") " pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.528244 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.528141    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:46.800464 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:46.800434    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"]
Sep 10 14:53:46.802306 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:53:46.802279    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b92655b_ac7f_45c4_b3de_3fbdc8dec5b8.slice/crio-1dd8540aa5178a46261daa6015e8bea1f6eb9c9efc31b707eaa11c974180a6be WatchSource:0}: Error finding container 1dd8540aa5178a46261daa6015e8bea1f6eb9c9efc31b707eaa11c974180a6be: Status 404 returned error can't find the container with id 1dd8540aa5178a46261daa6015e8bea1f6eb9c9efc31b707eaa11c974180a6be
Sep 10 14:53:47.665433 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:47.665384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh" event={"ID":"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8","Type":"ContainerStarted","Data":"9a31ebf15eb391ee43ef6a6011e2e6d4e5086e5f55771d1a17c6d721f081b95f"}
Sep 10 14:53:47.665433 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:47.665435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh" event={"ID":"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8","Type":"ContainerStarted","Data":"1dd8540aa5178a46261daa6015e8bea1f6eb9c9efc31b707eaa11c974180a6be"}
Sep 10 14:53:47.684476 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:47.684424    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh" podStartSLOduration=1.6844077259999999 podStartE2EDuration="1.684407726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:53:47.681776476 +0000 UTC m=+2045.873304674" watchObservedRunningTime="2026-09-10 14:53:47.684407726 +0000 UTC m=+2045.875935924"
Sep 10 14:53:56.528784 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:56.528739    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:56.529317 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:56.528802    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:56.541048 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:56.541019    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:53:56.725720 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:53:56.725691    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:54:04.797598 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:04.797506    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"]
Sep 10 14:54:06.522777 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:06.522744    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"]
Sep 10 14:54:19.819108 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:19.819063    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r" podUID="549350ab-b09e-41bb-8162-1046d9c4c263" containerName="main" containerID="cri-o://9e93548e109e4e04dde5129e6d2f5da78f4b7fd762809046f736e68c4f7e098a" gracePeriod=45
Sep 10 14:54:20.055412 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.055388    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:54:20.192018 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.191953    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-model-cache\") pod \"549350ab-b09e-41bb-8162-1046d9c4c263\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") "
Sep 10 14:54:20.192234 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.192039    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-tmp-dir\") pod \"549350ab-b09e-41bb-8162-1046d9c4c263\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") "
Sep 10 14:54:20.192234 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.192059    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-dshm\") pod \"549350ab-b09e-41bb-8162-1046d9c4c263\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") "
Sep 10 14:54:20.192234 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.192111    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm6qv\" (UniqueName: \"kubernetes.io/projected/549350ab-b09e-41bb-8162-1046d9c4c263-kube-api-access-zm6qv\") pod \"549350ab-b09e-41bb-8162-1046d9c4c263\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") "
Sep 10 14:54:20.192234 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.192132    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549350ab-b09e-41bb-8162-1046d9c4c263-tls-certs\") pod \"549350ab-b09e-41bb-8162-1046d9c4c263\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") "
Sep 10 14:54:20.192460 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.192270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-home\") pod \"549350ab-b09e-41bb-8162-1046d9c4c263\" (UID: \"549350ab-b09e-41bb-8162-1046d9c4c263\") "
Sep 10 14:54:20.192460 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.192274    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-model-cache" (OuterVolumeSpecName: "model-cache") pod "549350ab-b09e-41bb-8162-1046d9c4c263" (UID: "549350ab-b09e-41bb-8162-1046d9c4c263"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:20.192460 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.192376    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "549350ab-b09e-41bb-8162-1046d9c4c263" (UID: "549350ab-b09e-41bb-8162-1046d9c4c263"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:20.192625 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.192506    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-home" (OuterVolumeSpecName: "home") pod "549350ab-b09e-41bb-8162-1046d9c4c263" (UID: "549350ab-b09e-41bb-8162-1046d9c4c263"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:20.192704 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.192686    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:20.192766 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.192706    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:20.192766 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.192715    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:20.194252 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.194220    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-dshm" (OuterVolumeSpecName: "dshm") pod "549350ab-b09e-41bb-8162-1046d9c4c263" (UID: "549350ab-b09e-41bb-8162-1046d9c4c263"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:20.194252 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.194238    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/549350ab-b09e-41bb-8162-1046d9c4c263-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "549350ab-b09e-41bb-8162-1046d9c4c263" (UID: "549350ab-b09e-41bb-8162-1046d9c4c263"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:54:20.194441 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.194296    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549350ab-b09e-41bb-8162-1046d9c4c263-kube-api-access-zm6qv" (OuterVolumeSpecName: "kube-api-access-zm6qv") pod "549350ab-b09e-41bb-8162-1046d9c4c263" (UID: "549350ab-b09e-41bb-8162-1046d9c4c263"). InnerVolumeSpecName "kube-api-access-zm6qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:54:20.293517 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.293485    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549350ab-b09e-41bb-8162-1046d9c4c263-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:20.293517 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.293513    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm6qv\" (UniqueName: \"kubernetes.io/projected/549350ab-b09e-41bb-8162-1046d9c4c263-kube-api-access-zm6qv\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:20.293517 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.293525    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549350ab-b09e-41bb-8162-1046d9c4c263-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:20.846777 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.846746    2580 generic.go:356] "Generic (PLEG): container finished" podID="549350ab-b09e-41bb-8162-1046d9c4c263" containerID="9e93548e109e4e04dde5129e6d2f5da78f4b7fd762809046f736e68c4f7e098a" exitCode=0
Sep 10 14:54:20.847212 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.846786    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"
Sep 10 14:54:20.847212 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.846822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r" event={"ID":"549350ab-b09e-41bb-8162-1046d9c4c263","Type":"ContainerDied","Data":"9e93548e109e4e04dde5129e6d2f5da78f4b7fd762809046f736e68c4f7e098a"}
Sep 10 14:54:20.847212 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.846862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r" event={"ID":"549350ab-b09e-41bb-8162-1046d9c4c263","Type":"ContainerDied","Data":"be4e019dada971b1a1cc58a3cda07a4eb557cb24aabbfca18c31abb7e3763699"}
Sep 10 14:54:20.847212 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.846883    2580 scope.go:117] "RemoveContainer" containerID="9e93548e109e4e04dde5129e6d2f5da78f4b7fd762809046f736e68c4f7e098a"
Sep 10 14:54:20.855914 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.855895    2580 scope.go:117] "RemoveContainer" containerID="9e93548e109e4e04dde5129e6d2f5da78f4b7fd762809046f736e68c4f7e098a"
Sep 10 14:54:20.856221 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:54:20.856201    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e93548e109e4e04dde5129e6d2f5da78f4b7fd762809046f736e68c4f7e098a\": container with ID starting with 9e93548e109e4e04dde5129e6d2f5da78f4b7fd762809046f736e68c4f7e098a not found: ID does not exist" containerID="9e93548e109e4e04dde5129e6d2f5da78f4b7fd762809046f736e68c4f7e098a"
Sep 10 14:54:20.856279 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.856231    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e93548e109e4e04dde5129e6d2f5da78f4b7fd762809046f736e68c4f7e098a"} err="failed to get container status \"9e93548e109e4e04dde5129e6d2f5da78f4b7fd762809046f736e68c4f7e098a\": rpc error: code = NotFound desc = could not find container \"9e93548e109e4e04dde5129e6d2f5da78f4b7fd762809046f736e68c4f7e098a\": container with ID starting with 9e93548e109e4e04dde5129e6d2f5da78f4b7fd762809046f736e68c4f7e098a not found: ID does not exist"
Sep 10 14:54:20.863574 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.863550    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"]
Sep 10 14:54:20.865760 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:20.865740    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f681ae9c-kserve-66cf7c5b79-q558r"]
Sep 10 14:54:21.545142 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.545098    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh" podUID="8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8" containerName="main" containerID="cri-o://9a31ebf15eb391ee43ef6a6011e2e6d4e5086e5f55771d1a17c6d721f081b95f" gracePeriod=45
Sep 10 14:54:21.796818 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.796759    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:54:21.854528 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.854499    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8" containerID="9a31ebf15eb391ee43ef6a6011e2e6d4e5086e5f55771d1a17c6d721f081b95f" exitCode=0
Sep 10 14:54:21.854924 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.854541    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"
Sep 10 14:54:21.854924 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.854587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh" event={"ID":"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8","Type":"ContainerDied","Data":"9a31ebf15eb391ee43ef6a6011e2e6d4e5086e5f55771d1a17c6d721f081b95f"}
Sep 10 14:54:21.854924 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.854624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh" event={"ID":"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8","Type":"ContainerDied","Data":"1dd8540aa5178a46261daa6015e8bea1f6eb9c9efc31b707eaa11c974180a6be"}
Sep 10 14:54:21.854924 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.854645    2580 scope.go:117] "RemoveContainer" containerID="9a31ebf15eb391ee43ef6a6011e2e6d4e5086e5f55771d1a17c6d721f081b95f"
Sep 10 14:54:21.863436 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.863418    2580 scope.go:117] "RemoveContainer" containerID="9a31ebf15eb391ee43ef6a6011e2e6d4e5086e5f55771d1a17c6d721f081b95f"
Sep 10 14:54:21.863715 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:54:21.863696    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a31ebf15eb391ee43ef6a6011e2e6d4e5086e5f55771d1a17c6d721f081b95f\": container with ID starting with 9a31ebf15eb391ee43ef6a6011e2e6d4e5086e5f55771d1a17c6d721f081b95f not found: ID does not exist" containerID="9a31ebf15eb391ee43ef6a6011e2e6d4e5086e5f55771d1a17c6d721f081b95f"
Sep 10 14:54:21.863781 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.863727    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a31ebf15eb391ee43ef6a6011e2e6d4e5086e5f55771d1a17c6d721f081b95f"} err="failed to get container status \"9a31ebf15eb391ee43ef6a6011e2e6d4e5086e5f55771d1a17c6d721f081b95f\": rpc error: code = NotFound desc = could not find container \"9a31ebf15eb391ee43ef6a6011e2e6d4e5086e5f55771d1a17c6d721f081b95f\": container with ID starting with 9a31ebf15eb391ee43ef6a6011e2e6d4e5086e5f55771d1a17c6d721f081b95f not found: ID does not exist"
Sep 10 14:54:21.909951 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.909915    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4gvp\" (UniqueName: \"kubernetes.io/projected/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-kube-api-access-f4gvp\") pod \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") "
Sep 10 14:54:21.910123 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.909963    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-tmp-dir\") pod \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") "
Sep 10 14:54:21.910123 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.910007    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-dshm\") pod \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") "
Sep 10 14:54:21.910123 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.910058    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-tls-certs\") pod \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") "
Sep 10 14:54:21.910123 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.910087    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-home\") pod \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") "
Sep 10 14:54:21.910123 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.910110    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-model-cache\") pod \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\" (UID: \"8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8\") "
Sep 10 14:54:21.910388 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.910266    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8" (UID: "8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:21.910388 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.910368    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-home" (OuterVolumeSpecName: "home") pod "8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8" (UID: "8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:21.910511 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.910401    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8" (UID: "8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:21.910511 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.910458    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:21.910511 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.910473    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:21.912222 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.912203    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8" (UID: "8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:54:21.912616 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.912595    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-dshm" (OuterVolumeSpecName: "dshm") pod "8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8" (UID: "8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:54:21.912744 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:21.912619    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-kube-api-access-f4gvp" (OuterVolumeSpecName: "kube-api-access-f4gvp") pod "8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8" (UID: "8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8"). InnerVolumeSpecName "kube-api-access-f4gvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:54:22.011946 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:22.011906    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:22.011946 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:22.011936    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:22.011946 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:22.011945    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:22.011946 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:22.011955    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4gvp\" (UniqueName: \"kubernetes.io/projected/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8-kube-api-access-f4gvp\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:22.177247 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:22.177215    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"]
Sep 10 14:54:22.180504 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:22.180479    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f681ae9c-kserve-56f99dd98c-krqfh"]
Sep 10 14:54:22.310812 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:22.310779    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="549350ab-b09e-41bb-8162-1046d9c4c263" path="/var/lib/kubelet/pods/549350ab-b09e-41bb-8162-1046d9c4c263/volumes"
Sep 10 14:54:22.311317 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:22.311296    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8" path="/var/lib/kubelet/pods/8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8/volumes"
Sep 10 14:54:29.824402 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.824367    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"]
Sep 10 14:54:29.824777 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.824743    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="549350ab-b09e-41bb-8162-1046d9c4c263" containerName="main"
Sep 10 14:54:29.824777 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.824754    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="549350ab-b09e-41bb-8162-1046d9c4c263" containerName="main"
Sep 10 14:54:29.824777 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.824769    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8" containerName="main"
Sep 10 14:54:29.824777 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.824775    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8" containerName="main"
Sep 10 14:54:29.824923 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.824833    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b92655b-ac7f-45c4-b3de-3fbdc8dec5b8" containerName="main"
Sep 10 14:54:29.824923 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.824842    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="549350ab-b09e-41bb-8162-1046d9c4c263" containerName="main"
Sep 10 14:54:29.828579 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.828561    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:29.842029 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.841998    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"]
Sep 10 14:54:29.879160 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.879101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76-openshift-service-ca\") pod \"maas-api-54bf7886c5-zxsts\" (UID: \"e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76\") " pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:29.879326 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.879284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76-maas-api-tls\") pod \"maas-api-54bf7886c5-zxsts\" (UID: \"e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76\") " pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:29.879326 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.879303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76-metrics-certs\") pod \"maas-api-54bf7886c5-zxsts\" (UID: \"e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76\") " pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:29.879409 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.879337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnpv9\" (UniqueName: \"kubernetes.io/projected/e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76-kube-api-access-tnpv9\") pod \"maas-api-54bf7886c5-zxsts\" (UID: \"e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76\") " pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:29.980424 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.980379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnpv9\" (UniqueName: \"kubernetes.io/projected/e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76-kube-api-access-tnpv9\") pod \"maas-api-54bf7886c5-zxsts\" (UID: \"e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76\") " pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:29.980605 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.980456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76-openshift-service-ca\") pod \"maas-api-54bf7886c5-zxsts\" (UID: \"e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76\") " pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:29.980605 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.980544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76-maas-api-tls\") pod \"maas-api-54bf7886c5-zxsts\" (UID: \"e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76\") " pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:29.980605 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.980564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76-metrics-certs\") pod \"maas-api-54bf7886c5-zxsts\" (UID: \"e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76\") " pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:29.981227 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.981198    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76-openshift-service-ca\") pod \"maas-api-54bf7886c5-zxsts\" (UID: \"e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76\") " pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:29.983406 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.983368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76-maas-api-tls\") pod \"maas-api-54bf7886c5-zxsts\" (UID: \"e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76\") " pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:29.983406 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.983371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76-metrics-certs\") pod \"maas-api-54bf7886c5-zxsts\" (UID: \"e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76\") " pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:29.988483 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:29.988443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnpv9\" (UniqueName: \"kubernetes.io/projected/e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76-kube-api-access-tnpv9\") pod \"maas-api-54bf7886c5-zxsts\" (UID: \"e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76\") " pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:30.140291 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:30.140246    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:30.416073 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:30.416045    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"]
Sep 10 14:54:30.418119 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:54:30.418088    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58bc6ae_c0c0_4dcb_b78f_f1e52e9bfb76.slice/crio-9345b6ae1f9ceadbb5bdca91e80b39616fefed3f5d03865cff5e3c524bf4afc4 WatchSource:0}: Error finding container 9345b6ae1f9ceadbb5bdca91e80b39616fefed3f5d03865cff5e3c524bf4afc4: Status 404 returned error can't find the container with id 9345b6ae1f9ceadbb5bdca91e80b39616fefed3f5d03865cff5e3c524bf4afc4
Sep 10 14:54:30.903817 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:30.903787    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts" event={"ID":"e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76","Type":"ContainerStarted","Data":"9345b6ae1f9ceadbb5bdca91e80b39616fefed3f5d03865cff5e3c524bf4afc4"}
Sep 10 14:54:31.911250 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:31.911209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts" event={"ID":"e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76","Type":"ContainerStarted","Data":"1e64c21e022ce27bd701911dda1888006d4c395d53a3750c7a761434860d9ac0"}
Sep 10 14:54:31.911642 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:31.911335    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:31.928719 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:31.928661    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts" podStartSLOduration=2.473440359 podStartE2EDuration="2.928645383s" totalImagesPullingTime="455.205024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:54:31.927708744 +0000 UTC m=+2090.119236943" watchObservedRunningTime="2026-09-10 14:54:31.928645383 +0000 UTC m=+2090.120173580"
Sep 10 14:54:37.921870 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:37.921839    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54bf7886c5-zxsts"
Sep 10 14:54:42.458060 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:42.458021    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:54:42.471159 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:42.471132    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:54:45.743147 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:45.743107    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"]
Sep 10 14:54:45.743679 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:45.743418    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk" podUID="98a75836-b470-4200-96d0-717480b4ddbc" containerName="maas-api" containerID="cri-o://f5ba541811133ac26ee544c553a33e956bb7c5cf0ac9d96af1e7232f3fe34edf" gracePeriod=30
Sep 10 14:54:46.986857 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:46.986832    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:54:46.990613 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:46.990584    2580 generic.go:356] "Generic (PLEG): container finished" podID="98a75836-b470-4200-96d0-717480b4ddbc" containerID="f5ba541811133ac26ee544c553a33e956bb7c5cf0ac9d96af1e7232f3fe34edf" exitCode=0
Sep 10 14:54:46.990760 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:46.990619    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"
Sep 10 14:54:46.990760 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:46.990664    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk" event={"ID":"98a75836-b470-4200-96d0-717480b4ddbc","Type":"ContainerDied","Data":"f5ba541811133ac26ee544c553a33e956bb7c5cf0ac9d96af1e7232f3fe34edf"}
Sep 10 14:54:46.990760 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:46.990713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk" event={"ID":"98a75836-b470-4200-96d0-717480b4ddbc","Type":"ContainerDied","Data":"5e2626341c070105ea1ab447ae395bbb0c41c181b24ae48f1d1f355d268e6770"}
Sep 10 14:54:46.990760 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:46.990736    2580 scope.go:117] "RemoveContainer" containerID="f5ba541811133ac26ee544c553a33e956bb7c5cf0ac9d96af1e7232f3fe34edf"
Sep 10 14:54:47.000056 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.000038    2580 scope.go:117] "RemoveContainer" containerID="f5ba541811133ac26ee544c553a33e956bb7c5cf0ac9d96af1e7232f3fe34edf"
Sep 10 14:54:47.000359 ip-10-0-138-102 kubenswrapper[2580]: E0910 14:54:47.000339    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ba541811133ac26ee544c553a33e956bb7c5cf0ac9d96af1e7232f3fe34edf\": container with ID starting with f5ba541811133ac26ee544c553a33e956bb7c5cf0ac9d96af1e7232f3fe34edf not found: ID does not exist" containerID="f5ba541811133ac26ee544c553a33e956bb7c5cf0ac9d96af1e7232f3fe34edf"
Sep 10 14:54:47.000419 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.000369    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ba541811133ac26ee544c553a33e956bb7c5cf0ac9d96af1e7232f3fe34edf"} err="failed to get container status \"f5ba541811133ac26ee544c553a33e956bb7c5cf0ac9d96af1e7232f3fe34edf\": rpc error: code = NotFound desc = could not find container \"f5ba541811133ac26ee544c553a33e956bb7c5cf0ac9d96af1e7232f3fe34edf\": container with ID starting with f5ba541811133ac26ee544c553a33e956bb7c5cf0ac9d96af1e7232f3fe34edf not found: ID does not exist"
Sep 10 14:54:47.034216 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.034186    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a75836-b470-4200-96d0-717480b4ddbc-openshift-service-ca\") pod \"98a75836-b470-4200-96d0-717480b4ddbc\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") "
Sep 10 14:54:47.034401 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.034288    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a75836-b470-4200-96d0-717480b4ddbc-metrics-certs\") pod \"98a75836-b470-4200-96d0-717480b4ddbc\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") "
Sep 10 14:54:47.034401 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.034311    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz75w\" (UniqueName: \"kubernetes.io/projected/98a75836-b470-4200-96d0-717480b4ddbc-kube-api-access-hz75w\") pod \"98a75836-b470-4200-96d0-717480b4ddbc\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") "
Sep 10 14:54:47.034401 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.034357    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a75836-b470-4200-96d0-717480b4ddbc-maas-api-tls\") pod \"98a75836-b470-4200-96d0-717480b4ddbc\" (UID: \"98a75836-b470-4200-96d0-717480b4ddbc\") "
Sep 10 14:54:47.034579 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.034557    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a75836-b470-4200-96d0-717480b4ddbc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "98a75836-b470-4200-96d0-717480b4ddbc" (UID: "98a75836-b470-4200-96d0-717480b4ddbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:54:47.036515 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.036480    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a75836-b470-4200-96d0-717480b4ddbc-kube-api-access-hz75w" (OuterVolumeSpecName: "kube-api-access-hz75w") pod "98a75836-b470-4200-96d0-717480b4ddbc" (UID: "98a75836-b470-4200-96d0-717480b4ddbc"). InnerVolumeSpecName "kube-api-access-hz75w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:54:47.036656 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.036493    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a75836-b470-4200-96d0-717480b4ddbc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "98a75836-b470-4200-96d0-717480b4ddbc" (UID: "98a75836-b470-4200-96d0-717480b4ddbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:54:47.036656 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.036550    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a75836-b470-4200-96d0-717480b4ddbc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "98a75836-b470-4200-96d0-717480b4ddbc" (UID: "98a75836-b470-4200-96d0-717480b4ddbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:54:47.135782 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.135751    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a75836-b470-4200-96d0-717480b4ddbc-metrics-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:47.135782 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.135779    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz75w\" (UniqueName: \"kubernetes.io/projected/98a75836-b470-4200-96d0-717480b4ddbc-kube-api-access-hz75w\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:47.136014 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.135790    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a75836-b470-4200-96d0-717480b4ddbc-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:47.136014 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.135799    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a75836-b470-4200-96d0-717480b4ddbc-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\""
Sep 10 14:54:47.317903 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.317867    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"]
Sep 10 14:54:47.325323 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:47.325297    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk"]
Sep 10 14:54:48.310343 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:48.310309    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98a75836-b470-4200-96d0-717480b4ddbc" path="/var/lib/kubelet/pods/98a75836-b470-4200-96d0-717480b4ddbc/volumes"
Sep 10 14:54:49.972534 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:54:49.972476    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-f96f74b55-zrjbk" podUID="98a75836-b470-4200-96d0-717480b4ddbc" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.42:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 10 14:55:04.879085 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:55:04.879044    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6857cb7d9d-tmvsw_133328f1-4c9c-4289-b248-5e16dd22cdc0/manager/0.log"
Sep 10 14:55:05.232436 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:55:05.232407    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c47c8c9b-drn2t_69a116ca-0e8a-413a-9e47-3d78f6d18441/copy-manifests/0.log"
Sep 10 14:55:05.238637 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:55:05.238613    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c47c8c9b-drn2t_69a116ca-0e8a-413a-9e47-3d78f6d18441/manager/0.log"
Sep 10 14:55:15.939080 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:55:15.939046    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-54bf7886c5-zxsts_e58bc6ae-c0c0-4dcb-b78f-f1e52e9bfb76/maas-api/0.log"
Sep 10 14:55:37.412124 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:55:37.412051    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-cnwx8_5ea7525c-71f9-4bc1-875e-61659813fb09/kuadrant-console-plugin/0.log"
Sep 10 14:55:38.490538 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:55:38.490493    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79c7cbc6bf-jxtdf_6f820d02-66ad-4454-9b97-84fb994bbf71/kube-auth-proxy/0.log"
Sep 10 14:55:39.290410 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:55:39.290380    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-css28_36135db2-697e-40d0-9824-57128be4bf3a/main/0.log"
Sep 10 14:55:39.511875 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:55:39.511845    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-kmv4m_1c2fe6e5-69e2-4c53-aa13-912642cdab26/main/0.log"
Sep 10 14:55:39.948549 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:55:39.948518    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-cgpmf_22de848f-e5a4-4e08-8ae2-7fd2dc425c25/main/0.log"
Sep 10 14:56:05.045154 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.045115    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4xhnq/must-gather-hkz84"]
Sep 10 14:56:05.045750 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.045708    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a75836-b470-4200-96d0-717480b4ddbc" containerName="maas-api"
Sep 10 14:56:05.045750 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.045730    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a75836-b470-4200-96d0-717480b4ddbc" containerName="maas-api"
Sep 10 14:56:05.045889 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.045847    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a75836-b470-4200-96d0-717480b4ddbc" containerName="maas-api"
Sep 10 14:56:05.049377 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.049354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4xhnq/must-gather-hkz84"
Sep 10 14:56:05.051734 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.051707    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4xhnq\"/\"kube-root-ca.crt\""
Sep 10 14:56:05.052814 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.052791    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4xhnq\"/\"default-dockercfg-24sbf\""
Sep 10 14:56:05.052939 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.052804    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4xhnq\"/\"openshift-service-ca.crt\""
Sep 10 14:56:05.055892 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.055866    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4xhnq/must-gather-hkz84"]
Sep 10 14:56:05.147779 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.147739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6f4h\" (UniqueName: \"kubernetes.io/projected/54b6d902-9de7-4f50-b5c8-f9d7a84b3873-kube-api-access-m6f4h\") pod \"must-gather-hkz84\" (UID: \"54b6d902-9de7-4f50-b5c8-f9d7a84b3873\") " pod="openshift-must-gather-4xhnq/must-gather-hkz84"
Sep 10 14:56:05.148001 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.147803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/54b6d902-9de7-4f50-b5c8-f9d7a84b3873-must-gather-output\") pod \"must-gather-hkz84\" (UID: \"54b6d902-9de7-4f50-b5c8-f9d7a84b3873\") " pod="openshift-must-gather-4xhnq/must-gather-hkz84"
Sep 10 14:56:05.248878 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.248838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6f4h\" (UniqueName: \"kubernetes.io/projected/54b6d902-9de7-4f50-b5c8-f9d7a84b3873-kube-api-access-m6f4h\") pod \"must-gather-hkz84\" (UID: \"54b6d902-9de7-4f50-b5c8-f9d7a84b3873\") " pod="openshift-must-gather-4xhnq/must-gather-hkz84"
Sep 10 14:56:05.249088 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.248889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/54b6d902-9de7-4f50-b5c8-f9d7a84b3873-must-gather-output\") pod \"must-gather-hkz84\" (UID: \"54b6d902-9de7-4f50-b5c8-f9d7a84b3873\") " pod="openshift-must-gather-4xhnq/must-gather-hkz84"
Sep 10 14:56:05.249216 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.249200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/54b6d902-9de7-4f50-b5c8-f9d7a84b3873-must-gather-output\") pod \"must-gather-hkz84\" (UID: \"54b6d902-9de7-4f50-b5c8-f9d7a84b3873\") " pod="openshift-must-gather-4xhnq/must-gather-hkz84"
Sep 10 14:56:05.257608 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.257587    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6f4h\" (UniqueName: \"kubernetes.io/projected/54b6d902-9de7-4f50-b5c8-f9d7a84b3873-kube-api-access-m6f4h\") pod \"must-gather-hkz84\" (UID: \"54b6d902-9de7-4f50-b5c8-f9d7a84b3873\") " pod="openshift-must-gather-4xhnq/must-gather-hkz84"
Sep 10 14:56:05.359161 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.359059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4xhnq/must-gather-hkz84"
Sep 10 14:56:05.644866 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.638589    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4xhnq/must-gather-hkz84"]
Sep 10 14:56:05.644866 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:05.642754    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:56:06.412952 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:06.412692    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4xhnq/must-gather-hkz84" event={"ID":"54b6d902-9de7-4f50-b5c8-f9d7a84b3873","Type":"ContainerStarted","Data":"80315adb4d130472ff3ac5bbcd5fc6efc02bbfa151c4c80632271b2b8e9d0568"}
Sep 10 14:56:06.412952 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:06.412750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4xhnq/must-gather-hkz84" event={"ID":"54b6d902-9de7-4f50-b5c8-f9d7a84b3873","Type":"ContainerStarted","Data":"116d6e4bf413bb1c2afcf6f810ed3437962b2cddafd7145b8425d9be9ac801ac"}
Sep 10 14:56:07.423747 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:07.423698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4xhnq/must-gather-hkz84" event={"ID":"54b6d902-9de7-4f50-b5c8-f9d7a84b3873","Type":"ContainerStarted","Data":"10e5c48b04b8aa42f95c3695e0074d931233f88b4be443d0c3497e23585e3b46"}
Sep 10 14:56:07.441432 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:07.441384    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4xhnq/must-gather-hkz84" podStartSLOduration=1.793099913 podStartE2EDuration="2.441366632s" totalImagesPullingTime="648.266719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:56:07.440376251 +0000 UTC m=+2185.631904465" watchObservedRunningTime="2026-09-10 14:56:07.441366632 +0000 UTC m=+2185.632894829"
Sep 10 14:56:07.885433 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:07.885399    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hgkhl_0b046f46-d48b-4025-b9f9-37c5eea82ff9/global-pull-secret-syncer/0.log"
Sep 10 14:56:07.956304 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:07.956275    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n82cf_496deff1-b6df-408f-8282-2b61f4563584/konnectivity-agent/0.log"
Sep 10 14:56:08.040181 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:08.040151    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-102.ec2.internal_e416b0faba17117313669ffa5d00f48b/haproxy/0.log"
Sep 10 14:56:12.853408 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:12.853366    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-cnwx8_5ea7525c-71f9-4bc1-875e-61659813fb09/kuadrant-console-plugin/0.log"
Sep 10 14:56:14.457174 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:14.457131    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_563ccb51-a496-4cca-b591-a5ed88e98f4a/alertmanager/0.log"
Sep 10 14:56:14.476869 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:14.476830    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_563ccb51-a496-4cca-b591-a5ed88e98f4a/config-reloader/0.log"
Sep 10 14:56:14.497151 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:14.497099    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_563ccb51-a496-4cca-b591-a5ed88e98f4a/kube-rbac-proxy-web/0.log"
Sep 10 14:56:14.515252 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:14.515206    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_563ccb51-a496-4cca-b591-a5ed88e98f4a/kube-rbac-proxy/0.log"
Sep 10 14:56:14.546723 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:14.546697    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_563ccb51-a496-4cca-b591-a5ed88e98f4a/kube-rbac-proxy-metric/0.log"
Sep 10 14:56:14.589475 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:14.589447    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_563ccb51-a496-4cca-b591-a5ed88e98f4a/prom-label-proxy/0.log"
Sep 10 14:56:14.607715 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:14.607663    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_563ccb51-a496-4cca-b591-a5ed88e98f4a/init-config-reloader/0.log"
Sep 10 14:56:14.756701 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:14.756616    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-nvfxb_837d6172-d2dd-43e7-90d2-ab57eb64d599/monitoring-plugin/0.log"
Sep 10 14:56:14.853372 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:14.853336    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hdwms_554aeb55-0a6e-4235-9490-5b9b7d7ddf55/node-exporter/0.log"
Sep 10 14:56:14.871647 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:14.871620    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hdwms_554aeb55-0a6e-4235-9490-5b9b7d7ddf55/kube-rbac-proxy/0.log"
Sep 10 14:56:14.893119 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:14.893071    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hdwms_554aeb55-0a6e-4235-9490-5b9b7d7ddf55/init-textfile/0.log"
Sep 10 14:56:15.019610 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:15.019573    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-bkwlf_88218fe6-c5fe-4431-b745-4fb80500b3d5/kube-rbac-proxy-main/0.log"
Sep 10 14:56:15.045998 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:15.045957    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-bkwlf_88218fe6-c5fe-4431-b745-4fb80500b3d5/kube-rbac-proxy-self/0.log"
Sep 10 14:56:15.065009 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:15.064945    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-bkwlf_88218fe6-c5fe-4431-b745-4fb80500b3d5/openshift-state-metrics/0.log"
Sep 10 14:56:15.268704 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:15.268620    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-fsffg_ca4a60bc-3ca1-4a04-a8a5-fff546d7f370/prometheus-operator/0.log"
Sep 10 14:56:15.283646 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:15.283617    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-fsffg_ca4a60bc-3ca1-4a04-a8a5-fff546d7f370/kube-rbac-proxy/0.log"
Sep 10 14:56:15.342656 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:15.342621    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bf9f95fc5-6bgxv_e8413b80-c9a4-4ce9-be64-0c02d31741d1/telemeter-client/0.log"
Sep 10 14:56:15.361760 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:15.361733    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bf9f95fc5-6bgxv_e8413b80-c9a4-4ce9-be64-0c02d31741d1/reload/0.log"
Sep 10 14:56:15.379456 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:15.379428    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bf9f95fc5-6bgxv_e8413b80-c9a4-4ce9-be64-0c02d31741d1/kube-rbac-proxy/0.log"
Sep 10 14:56:15.409879 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:15.409847    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78967d786b-nxr8w_f8ceed02-deb5-4069-abfc-b53ea2636458/thanos-query/0.log"
Sep 10 14:56:15.434309 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:15.434275    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78967d786b-nxr8w_f8ceed02-deb5-4069-abfc-b53ea2636458/kube-rbac-proxy-web/0.log"
Sep 10 14:56:15.452436 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:15.452371    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78967d786b-nxr8w_f8ceed02-deb5-4069-abfc-b53ea2636458/kube-rbac-proxy/0.log"
Sep 10 14:56:15.471760 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:15.471728    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78967d786b-nxr8w_f8ceed02-deb5-4069-abfc-b53ea2636458/prom-label-proxy/0.log"
Sep 10 14:56:15.495898 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:15.495825    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78967d786b-nxr8w_f8ceed02-deb5-4069-abfc-b53ea2636458/kube-rbac-proxy-rules/0.log"
Sep 10 14:56:15.515578 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:15.515542    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78967d786b-nxr8w_f8ceed02-deb5-4069-abfc-b53ea2636458/kube-rbac-proxy-metrics/0.log"
Sep 10 14:56:18.250546 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.250495    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"]
Sep 10 14:56:18.255355 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.255330    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.265068 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.265041    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"]
Sep 10 14:56:18.291999 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.291932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e538106-c55f-4898-9d67-1b3c8e794925-lib-modules\") pod \"perf-node-gather-daemonset-6wrdk\" (UID: \"3e538106-c55f-4898-9d67-1b3c8e794925\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.292333 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.292310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wz5l\" (UniqueName: \"kubernetes.io/projected/3e538106-c55f-4898-9d67-1b3c8e794925-kube-api-access-8wz5l\") pod \"perf-node-gather-daemonset-6wrdk\" (UID: \"3e538106-c55f-4898-9d67-1b3c8e794925\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.292490 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.292475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e538106-c55f-4898-9d67-1b3c8e794925-proc\") pod \"perf-node-gather-daemonset-6wrdk\" (UID: \"3e538106-c55f-4898-9d67-1b3c8e794925\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.292609 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.292596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e538106-c55f-4898-9d67-1b3c8e794925-sys\") pod \"perf-node-gather-daemonset-6wrdk\" (UID: \"3e538106-c55f-4898-9d67-1b3c8e794925\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.292724 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.292712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e538106-c55f-4898-9d67-1b3c8e794925-podres\") pod \"perf-node-gather-daemonset-6wrdk\" (UID: \"3e538106-c55f-4898-9d67-1b3c8e794925\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.393944 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.393905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e538106-c55f-4898-9d67-1b3c8e794925-proc\") pod \"perf-node-gather-daemonset-6wrdk\" (UID: \"3e538106-c55f-4898-9d67-1b3c8e794925\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.394228 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.394208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e538106-c55f-4898-9d67-1b3c8e794925-sys\") pod \"perf-node-gather-daemonset-6wrdk\" (UID: \"3e538106-c55f-4898-9d67-1b3c8e794925\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.394349 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.394336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e538106-c55f-4898-9d67-1b3c8e794925-podres\") pod \"perf-node-gather-daemonset-6wrdk\" (UID: \"3e538106-c55f-4898-9d67-1b3c8e794925\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.394519 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.394505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e538106-c55f-4898-9d67-1b3c8e794925-lib-modules\") pod \"perf-node-gather-daemonset-6wrdk\" (UID: \"3e538106-c55f-4898-9d67-1b3c8e794925\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.394685 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.394670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wz5l\" (UniqueName: \"kubernetes.io/projected/3e538106-c55f-4898-9d67-1b3c8e794925-kube-api-access-8wz5l\") pod \"perf-node-gather-daemonset-6wrdk\" (UID: \"3e538106-c55f-4898-9d67-1b3c8e794925\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.395214 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.395192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e538106-c55f-4898-9d67-1b3c8e794925-proc\") pod \"perf-node-gather-daemonset-6wrdk\" (UID: \"3e538106-c55f-4898-9d67-1b3c8e794925\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.395365 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.395319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e538106-c55f-4898-9d67-1b3c8e794925-podres\") pod \"perf-node-gather-daemonset-6wrdk\" (UID: \"3e538106-c55f-4898-9d67-1b3c8e794925\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.395473 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.395350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e538106-c55f-4898-9d67-1b3c8e794925-sys\") pod \"perf-node-gather-daemonset-6wrdk\" (UID: \"3e538106-c55f-4898-9d67-1b3c8e794925\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.395580 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.395482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e538106-c55f-4898-9d67-1b3c8e794925-lib-modules\") pod \"perf-node-gather-daemonset-6wrdk\" (UID: \"3e538106-c55f-4898-9d67-1b3c8e794925\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.412211 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.412178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wz5l\" (UniqueName: \"kubernetes.io/projected/3e538106-c55f-4898-9d67-1b3c8e794925-kube-api-access-8wz5l\") pod \"perf-node-gather-daemonset-6wrdk\" (UID: \"3e538106-c55f-4898-9d67-1b3c8e794925\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.575309 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.575226    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:18.885664 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:18.885621    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"]
Sep 10 14:56:18.886775 ip-10-0-138-102 kubenswrapper[2580]: W0910 14:56:18.886737    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3e538106_c55f_4898_9d67_1b3c8e794925.slice/crio-23c606aa6f2d5f847f462e1559e8c23bed299800b5352832dcc05597240852af WatchSource:0}: Error finding container 23c606aa6f2d5f847f462e1559e8c23bed299800b5352832dcc05597240852af: Status 404 returned error can't find the container with id 23c606aa6f2d5f847f462e1559e8c23bed299800b5352832dcc05597240852af
Sep 10 14:56:19.531761 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:19.531712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk" event={"ID":"3e538106-c55f-4898-9d67-1b3c8e794925","Type":"ContainerStarted","Data":"9e0d3274e4cb06b2c4ddfcc99f86e9ffc256022cdedc0c984b5aa827b9524db3"}
Sep 10 14:56:19.532300 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:19.531791    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk" event={"ID":"3e538106-c55f-4898-9d67-1b3c8e794925","Type":"ContainerStarted","Data":"23c606aa6f2d5f847f462e1559e8c23bed299800b5352832dcc05597240852af"}
Sep 10 14:56:19.533045 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:19.533015    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:20.126519 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:20.126487    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ss76l_e8bdbb9e-6d22-4d3e-af97-c34159145a29/dns/0.log"
Sep 10 14:56:20.146078 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:20.146030    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ss76l_e8bdbb9e-6d22-4d3e-af97-c34159145a29/kube-rbac-proxy/0.log"
Sep 10 14:56:20.207186 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:20.207157    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vjqdq_48aab852-7724-4969-b2c7-d6d2db10ec8d/dns-node-resolver/0.log"
Sep 10 14:56:20.790665 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:20.790640    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4bwvd_3523580c-458a-486f-9cc6-8705298c0617/node-ca/0.log"
Sep 10 14:56:21.956919 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:21.956884    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79c7cbc6bf-jxtdf_6f820d02-66ad-4454-9b97-84fb994bbf71/kube-auth-proxy/0.log"
Sep 10 14:56:22.962350 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:22.962323    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6z7bn_bd2cae1f-0990-4824-857c-a1872793416d/serve-healthcheck-canary/0.log"
Sep 10 14:56:23.574097 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:23.574071    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wcprc_2b0aadca-c96f-4125-addf-c701034d2576/kube-rbac-proxy/0.log"
Sep 10 14:56:23.597176 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:23.597149    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wcprc_2b0aadca-c96f-4125-addf-c701034d2576/exporter/0.log"
Sep 10 14:56:23.621324 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:23.621295    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wcprc_2b0aadca-c96f-4125-addf-c701034d2576/extractor/0.log"
Sep 10 14:56:25.910717 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:25.910671    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6857cb7d9d-tmvsw_133328f1-4c9c-4289-b248-5e16dd22cdc0/manager/0.log"
Sep 10 14:56:26.021417 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:26.021385    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c47c8c9b-drn2t_69a116ca-0e8a-413a-9e47-3d78f6d18441/manager/0.log"
Sep 10 14:56:26.038369 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:26.038337    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c47c8c9b-drn2t_69a116ca-0e8a-413a-9e47-3d78f6d18441/copy-manifests/0.log"
Sep 10 14:56:26.557996 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:26.557948    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk"
Sep 10 14:56:26.574593 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:26.574522    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-6wrdk" podStartSLOduration=8.574499938 podStartE2EDuration="8.574499938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:56:19.551642906 +0000 UTC m=+2197.743171105" watchObservedRunningTime="2026-09-10 14:56:26.574499938 +0000 UTC m=+2204.766028137"
Sep 10 14:56:27.293943 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:27.293915    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-759b87488b-n8mk6_a0d93290-18b4-49c9-b462-8c8c219ddf77/manager/0.log"
Sep 10 14:56:28.496209 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:28.496098    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79c7cbc6bf-jxtdf_6f820d02-66ad-4454-9b97-84fb994bbf71/kube-auth-proxy/0.log"
Sep 10 14:56:30.444476 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:30.444444    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79c7cbc6bf-jxtdf_6f820d02-66ad-4454-9b97-84fb994bbf71/kube-auth-proxy/0.log"
Sep 10 14:56:33.895506 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:33.895477    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-47x2k_20919d23-b9fc-4552-8082-4b83053f870b/kube-multus/0.log"
Sep 10 14:56:33.957428 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:33.957401    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nf6ht_51965344-8fbc-41c8-b3ef-2f553017edbe/kube-multus-additional-cni-plugins/0.log"
Sep 10 14:56:33.976499 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:33.976474    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nf6ht_51965344-8fbc-41c8-b3ef-2f553017edbe/egress-router-binary-copy/0.log"
Sep 10 14:56:34.004743 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:34.004715    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nf6ht_51965344-8fbc-41c8-b3ef-2f553017edbe/cni-plugins/0.log"
Sep 10 14:56:34.021805 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:34.021781    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nf6ht_51965344-8fbc-41c8-b3ef-2f553017edbe/bond-cni-plugin/0.log"
Sep 10 14:56:34.038122 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:34.038098    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nf6ht_51965344-8fbc-41c8-b3ef-2f553017edbe/routeoverride-cni/0.log"
Sep 10 14:56:34.055144 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:34.055117    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nf6ht_51965344-8fbc-41c8-b3ef-2f553017edbe/whereabouts-cni-bincopy/0.log"
Sep 10 14:56:34.077300 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:34.077275    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nf6ht_51965344-8fbc-41c8-b3ef-2f553017edbe/whereabouts-cni/0.log"
Sep 10 14:56:34.458851 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:34.458816    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qvxlq_50de453c-4d08-4f49-9e7e-0d5e2221e72e/network-metrics-daemon/0.log"
Sep 10 14:56:34.471601 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:34.471574    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qvxlq_50de453c-4d08-4f49-9e7e-0d5e2221e72e/kube-rbac-proxy/0.log"
Sep 10 14:56:35.555087 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:35.555047    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-controller/0.log"
Sep 10 14:56:35.569548 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:35.569516    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/1.log"
Sep 10 14:56:35.569548 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:35.569528    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovn-acl-logging/0.log"
Sep 10 14:56:35.586015 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:35.585988    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/kube-rbac-proxy-node/0.log"
Sep 10 14:56:35.603865 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:35.603833    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/kube-rbac-proxy-ovn-metrics/0.log"
Sep 10 14:56:35.624872 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:35.624846    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/northd/0.log"
Sep 10 14:56:35.645785 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:35.645756    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/nbdb/0.log"
Sep 10 14:56:35.663493 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:35.663470    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/sbdb/0.log"
Sep 10 14:56:35.725005 ip-10-0-138-102 kubenswrapper[2580]: I0910 14:56:35.724956    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rjjh6_a7ef5dcd-b063-44f5-85ae-d9dc78521768/ovnkube-controller/0.log"