Sep 04 23:36:26.255641 ip-10-0-138-66 systemd[1]: Starting Kubernetes Kubelet...
Sep 04 23:36:26.699290 ip-10-0-138-66 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 04 23:36:26.699290 ip-10-0-138-66 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 04 23:36:26.699290 ip-10-0-138-66 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 04 23:36:26.699290 ip-10-0-138-66 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 04 23:36:26.699959 ip-10-0-138-66 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 04 23:36:26.701245 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.701132    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 04 23:36:26.704952 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704937    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 23:36:26.704952 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704953    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704957    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704960    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704964    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704968    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704971    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704974    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704977    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704980    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704983    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704986    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704989    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704992    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704995    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.704998    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705000    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705003    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705005    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705008    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 23:36:26.705016 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705010    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705013    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705015    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705018    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705022    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705025    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705028    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705031    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705033    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705036    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705039    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705041    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705044    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705046    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705049    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705051    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705054    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705056    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705061    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 23:36:26.705559 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705064    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705067    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705070    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705073    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705075    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705078    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705081    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705084    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705086    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705089    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705091    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705095    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705098    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705101    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705104    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705109    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705112    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705115    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705118    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 23:36:26.706188 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705120    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705124    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705126    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705129    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705132    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705134    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705137    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705140    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705143    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705145    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705148    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705150    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705153    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705155    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705159    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705163    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705165    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705168    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705170    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705173    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 23:36:26.706839 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705176    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705179    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705181    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705184    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705186    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705189    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705191    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705194    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705197    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705202    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705204    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705207    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705210    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705212    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705215    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705218    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705220    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705223    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705226    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705228    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 23:36:26.707427 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.705231    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707423    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707433    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707436    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707439    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707442    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707445    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707448    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707450    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707453    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707456    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707459    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707462    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707465    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707467    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707470    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707473    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707475    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707478    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707480    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 23:36:26.707914 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707483    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707486    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707488    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707491    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707493    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707496    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707499    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707501    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707504    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707506    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707509    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707512    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707514    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707516    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707520    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707522    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707525    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707528    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707530    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707533    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 23:36:26.708536 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707536    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707538    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707541    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707544    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707546    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707549    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707551    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707554    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707556    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707559    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707562    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707565    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707567    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707570    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707572    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707575    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707577    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707580    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707583    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707585    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 23:36:26.709054 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707588    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707592    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707596    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707600    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707603    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707606    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707610    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707613    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707616    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707618    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707621    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707624    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707627    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707630    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707632    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707635    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707638    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707640    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707643    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 23:36:26.709571 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707646    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707648    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707650    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707654    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707658    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707662    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707665    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707668    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707670    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707673    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707675    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707678    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707680    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707684    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707687    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707689    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707692    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707694    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707697    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 23:36:26.710037 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707700    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.707703    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708064    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708068    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708071    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708074    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708077    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708080    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708082    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708085    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708087    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708089    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708092    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708095    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708097    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708100    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708102    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708105    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708108    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 23:36:26.710508 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708111    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708115    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708119    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708122    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708125    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708128    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708130    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708132    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708135    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708138    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708140    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708142    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708145    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708147    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708150    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708153    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708155    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708158    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708160    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 23:36:26.710967 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708163    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708165    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708168    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708170    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708173    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708176    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708178    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708180    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708183    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708185    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708188    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708190    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708193    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708196    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708199    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708201    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708204    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708207    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708210    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708212    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 23:36:26.711442 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708215    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708217    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708220    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708222    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708225    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708228    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708231    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708234    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708237    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708239    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708242    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708244    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708247    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708249    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708252    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708257    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708260    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708262    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708265    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708268    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 23:36:26.711960 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708270    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708273    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708276    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708278    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708280    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708283    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708285    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708288    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708290    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708293    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708295    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708298    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708300    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708304    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708307    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708310    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708314    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708317    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708320    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708323    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 23:36:26.712447 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708326    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708329    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.708331    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708423    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708457    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708463    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708468    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708473    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708477    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708481    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708485    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708488    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708491    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708495    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708498    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708501    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708504    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708507    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708510    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708514    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708517    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708522    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708525    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 04 23:36:26.712931 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708529    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708532    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708535    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708539    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708542    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708545    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708548    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708553    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708556    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708559    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708562    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708565    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708569    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708572    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708575    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708578    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708582    2562 flags.go:64] FLAG: --enable-server="true"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708585    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708589    2562 flags.go:64] FLAG: --event-burst="100"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708593    2562 flags.go:64] FLAG: --event-qps="50"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708595    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708598    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708602    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708605    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 04 23:36:26.713549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708608    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708611    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708614    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708616    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708619    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708623    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708626    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708629    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708632    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708634    2562 flags.go:64] FLAG: --feature-gates=""
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708638    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708641    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708644    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708646    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708649    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708654    2562 flags.go:64] FLAG: --help="false"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708656    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-138-66.ec2.internal"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708659    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708662    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708665    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708668    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708672    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708674    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 04 23:36:26.714146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708677    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708680    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708683    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708686    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708689    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708692    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708695    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708698    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708701    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708703    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708706    2562 flags.go:64] FLAG: --lock-file=""
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708709    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708712    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708715    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708720    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708723    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708726    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708729    2562 flags.go:64] FLAG: --logging-format="text"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708732    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708735    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708738    2562 flags.go:64] FLAG: --manifest-url=""
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708740    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708745    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708747    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708752    2562 flags.go:64] FLAG: --max-pods="110"
Sep 04 23:36:26.714735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708755    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708758    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708761    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708764    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708767    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708770    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708772    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708780    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708783    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708785    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708788    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708791    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708797    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708799    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708802    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708805    2562 flags.go:64] FLAG: --port="10250"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708808    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708811    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b2f3424db9c05718"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708814    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708817    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708820    2562 flags.go:64] FLAG: --register-node="true"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708823    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708827    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708830    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 04 23:36:26.715387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708833    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708836    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708839    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708842    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708845    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708848    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708851    2562 flags.go:64] FLAG: --runonce="false"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708853    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708858    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708862    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708864    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708867    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708870    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708873    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708876    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708879    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708882    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708884    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708887    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708890    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708893    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708896    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708903    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708907    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708912    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 04 23:36:26.715970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708919    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708925    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708929    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708933    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708938    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708941    2562 flags.go:64] FLAG: --v="2"
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708951    2562 flags.go:64] FLAG: --version="false"
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708959    2562 flags.go:64] FLAG: --vmodule=""
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708963    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.708966    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709084    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709087    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709091    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709094    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709097    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709101    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709103    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709106    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709108    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709111    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709113    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709116    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 23:36:26.716609 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709119    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709121    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709124    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709126    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709130    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709134    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709137    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709140    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709143    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709146    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709148    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709152    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709155    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709157    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709161    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709164    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709167    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709170    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709178    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 23:36:26.717148 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709181    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709183    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709186    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709189    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709191    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709194    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709198    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709200    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709203    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709205    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709208    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709210    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709213    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709215    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709218    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709221    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709223    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709226    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709229    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709231    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 23:36:26.717629 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709233    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709236    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709239    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709241    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709245    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709249    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709252    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709254    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709257    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709259    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709262    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709265    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709273    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709275    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709278    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709280    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709283    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709285    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709289    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709292    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 23:36:26.718109 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709295    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709297    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709300    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709302    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709305    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709307    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709310    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709313    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709315    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709318    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709320    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709322    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709325    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709328    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709331    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709333    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709337    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709340    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709343    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709345    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 23:36:26.718614 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709348    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 23:36:26.719114 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709350    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 23:36:26.719114 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709353    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 23:36:26.719114 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709356    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 23:36:26.719114 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709358    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 23:36:26.719114 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709361    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 23:36:26.719114 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709364    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 23:36:26.719114 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.709380    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 23:36:26.719114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.710024    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 04 23:36:26.719114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.717453    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 04 23:36:26.719114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.717468    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 04 23:36:26.719114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.718257    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 04 23:36:26.719114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.718264    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 04 23:36:26.719114 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718338    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 23:36:26.719114 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718345    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 23:36:26.719114 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718348    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718351    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718353    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718356    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718359    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718361    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718364    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718382    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718385    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718388    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718391    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718394    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718397    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718399    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718403    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718407    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718410    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718413    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718416    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 23:36:26.719522 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718418    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718421    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718423    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718426    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718429    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718431    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718434    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718437    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718446    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718449    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718451    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718454    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718457    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718459    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718462    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718464    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718467    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718470    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718472    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718475    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 23:36:26.719983 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718478    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718481    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718483    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718486    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718489    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718491    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718494    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718496    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718499    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718502    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718504    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718507    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718511    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718514    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718517    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718520    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718523    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718526    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718528    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 23:36:26.720474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718531    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718534    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718538    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718541    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718543    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718546    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718548    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718551    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718553    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718556    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718558    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718561    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718563    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718565    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718569    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718573    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718576    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718579    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718583    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 23:36:26.720945 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718586    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718589    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718592    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718595    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718597    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718600    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718602    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718605    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718608    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718610    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718613    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718615    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718618    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718621    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718625    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718627    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718630    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718633    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718636    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 23:36:26.721418 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718638    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 23:36:26.721888 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.718643    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 04 23:36:26.721888 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718768    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 23:36:26.721888 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718774    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 23:36:26.721888 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718779    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 23:36:26.721888 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718782    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 23:36:26.721888 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718785    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 23:36:26.721888 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718788    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 23:36:26.721888 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718790    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 23:36:26.721888 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718793    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 23:36:26.721888 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718795    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 23:36:26.721888 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718798    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 23:36:26.721888 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718800    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 23:36:26.721888 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718803    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 23:36:26.721888 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718806    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718808    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718811    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718814    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718816    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718819    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718821    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718824    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718826    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718829    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718831    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718834    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718837    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718842    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718845    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718847    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718850    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718853    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718856    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718859    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 23:36:26.722230 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718861    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718863    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718866    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718868    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718871    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718874    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718876    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718878    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718881    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718883    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718886    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718888    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718891    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718894    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718896    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718899    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718901    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718904    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718906    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718909    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 23:36:26.722720 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718911    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718914    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718916    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718919    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718921    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718924    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718927    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718929    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718931    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718935    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718937    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718940    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718942    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718944    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718947    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718950    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718952    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718954    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718957    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718959    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 23:36:26.723187 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718962    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718964    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718967    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718969    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718972    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718974    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718977    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718980    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718982    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718985    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718987    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718990    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718992    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718995    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.718997    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.719000    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.719002    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.719005    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.719008    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 23:36:26.723812 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.719010    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 23:36:26.724297 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.719013    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 23:36:26.724297 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.719015    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 23:36:26.724297 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.719018    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 23:36:26.724297 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.719021    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 23:36:26.724297 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.719024    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 23:36:26.724297 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.719028    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 23:36:26.724297 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:26.719031    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 23:36:26.724297 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.719036    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 04 23:36:26.724297 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.719140    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 04 23:36:26.724297 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.721411    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 04 23:36:26.724297 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.722390    2562 server.go:1019] "Starting client certificate rotation"
Sep 04 23:36:26.724297 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.722496    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 04 23:36:26.724297 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.722528    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 04 23:36:26.725090 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.725077    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 04 23:36:26.726404 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.726388    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 04 23:36:26.735489 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.735390    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 04 23:36:26.738255 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.738240    2562 log.go:25] "Validated CRI v1 image API"
Sep 04 23:36:26.739129 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.739112    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 04 23:36:26.743221 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.743199    2562 fs.go:135] Filesystem UUIDs: map[20370aa2-3b12-4294-bb08-afc44615db8a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ee7f639f-275e-4d84-bd4d-135141be8886:/dev/nvme0n1p3]
Sep 04 23:36:26.743302 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.743219    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 04 23:36:26.748944 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.748820    2562 manager.go:216] Machine: {Timestamp:2026-09-04 23:36:26.747670106 +0000 UTC m=+0.367619156 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096473 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e2d492f7c4e4174d5b9c872515419 SystemUUID:ec2e2d49-2f7c-4e41-74d5-b9c872515419 BootID:2a39cc00-8034-422b-a72d-059cd7d46f51 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f2:eb:e6:af:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f2:eb:e6:af:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:d5:33:84:dd:3d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 04 23:36:26.748944 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.748934    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 04 23:36:26.749098 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.749050    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 04 23:36:26.752492 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.752457    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 04 23:36:26.752639 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.752492    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-66.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 04 23:36:26.752874 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.752857    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 04 23:36:26.753278 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.753267    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 04 23:36:26.753313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.753281    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 04 23:36:26.753313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.753289    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 04 23:36:26.754082 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.754071    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 04 23:36:26.755119 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.755107    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 04 23:36:26.759145 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.759134    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 04 23:36:26.759261 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.759252    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 04 23:36:26.759317 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.759281    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 04 23:36:26.759317 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.759291    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 04 23:36:26.759317 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.759309    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 04 23:36:26.759472 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.759379    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 04 23:36:26.759472 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.759400    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 04 23:36:26.759857 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.759844    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 04 23:36:26.759923 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.759865    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 04 23:36:26.762900 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.762883    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 04 23:36:26.766436 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.766415    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 04 23:36:26.766635 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.766610    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 04 23:36:26.768919 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.768900    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 04 23:36:26.770967 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.770948    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 04 23:36:26.770967 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.770967    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 04 23:36:26.771075 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.770973    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 04 23:36:26.771075 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.770984    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 04 23:36:26.771075 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.770992    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 04 23:36:26.771075 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.771011    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 04 23:36:26.771075 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.771018    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 04 23:36:26.771075 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.771030    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 04 23:36:26.771075 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.771038    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 04 23:36:26.771075 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.771047    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 04 23:36:26.771075 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.771052    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 04 23:36:26.771075 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.771081    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 04 23:36:26.771436 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.771089    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 04 23:36:26.771436 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.771133    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 04 23:36:26.771436 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.771160    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 04 23:36:26.773668 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.773651    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-66.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 04 23:36:26.775031 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.775019    2562 server.go:1268] "Started kubelet"
Sep 04 23:36:26.775125 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.775089    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 04 23:36:26.775187 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.775110    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 04 23:36:26.775233 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.775184    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 04 23:36:26.775924 ip-10-0-138-66 systemd[1]: Started Kubernetes Kubelet.
Sep 04 23:36:26.777439 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.777336    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 04 23:36:26.778338 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.778299    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 04 23:36:26.782149 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.782130    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 04 23:36:26.785534 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.785513    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 04 23:36:26.786356 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.786339    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 04 23:36:26.786540 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.786489    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-66.ec2.internal\" not found"
Sep 04 23:36:26.787248 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.787232    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 04 23:36:26.787248 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.787248    2562 factory.go:55] Registering systemd factory
Sep 04 23:36:26.787399 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.787257    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 04 23:36:26.787399 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.787261    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 04 23:36:26.787399 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.787265    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 04 23:36:26.787399 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.787288    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 04 23:36:26.787399 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.787390    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 04 23:36:26.787399 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.787401    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 04 23:36:26.787626 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.787504    2562 factory.go:153] Registering CRI-O factory
Sep 04 23:36:26.787626 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.787514    2562 factory.go:223] Registration of the crio container factory successfully
Sep 04 23:36:26.787626 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.787526    2562 factory.go:103] Registering Raw factory
Sep 04 23:36:26.787626 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.787536    2562 manager.go:1196] Started watching for new ooms in manager
Sep 04 23:36:26.787956 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.787943    2562 manager.go:349] Starting recovery of all containers
Sep 04 23:36:26.789224 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.789184    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 04 23:36:26.789309 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.789290    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 04 23:36:26.790355 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.789155    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-66.ec2.internal.18d24261166ca2d7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-66.ec2.internal,UID:ip-10-0-138-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-66.ec2.internal,},FirstTimestamp:2026-09-04 23:36:26.774987479 +0000 UTC m=+0.394936528,LastTimestamp:2026-09-04 23:36:26.774987479 +0000 UTC m=+0.394936528,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-66.ec2.internal,}"
Sep 04 23:36:26.799388 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.799355    2562 manager.go:354] Recovery completed
Sep 04 23:36:26.801247 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.801226    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w85fk"
Sep 04 23:36:26.804595 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.804584    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 23:36:26.806560 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.806540    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 23:36:26.806629 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.806565    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w85fk"
Sep 04 23:36:26.806629 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.806575    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 23:36:26.806629 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.806591    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasSufficientPID"
Sep 04 23:36:26.807150 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.807136    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 23:36:26.807223 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.807136    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 04 23:36:26.807223 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.807167    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 04 23:36:26.807223 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.807186    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 04 23:36:26.807853 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.807832    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 23:36:26.807913 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.807872    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 23:36:26.807913 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.807890    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasSufficientPID"
Sep 04 23:36:26.808445 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.808358    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-66.ec2.internal.18d24261184e61c9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-66.ec2.internal,UID:ip-10-0-138-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-66.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-66.ec2.internal,},FirstTimestamp:2026-09-04 23:36:26.806559177 +0000 UTC m=+0.426508233,LastTimestamp:2026-09-04 23:36:26.806559177 +0000 UTC m=+0.426508233,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-66.ec2.internal,}"
Sep 04 23:36:26.817450 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.817340    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-66.ec2.internal.18d24261184ec04d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-66.ec2.internal,UID:ip-10-0-138-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-66.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-66.ec2.internal,},FirstTimestamp:2026-09-04 23:36:26.806583373 +0000 UTC m=+0.426532421,LastTimestamp:2026-09-04 23:36:26.806583373 +0000 UTC m=+0.426532421,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-66.ec2.internal,}"
Sep 04 23:36:26.819867 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.819794    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-66.ec2.internal.18d24261184ef0b0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-66.ec2.internal,UID:ip-10-0-138-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-66.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-66.ec2.internal,},FirstTimestamp:2026-09-04 23:36:26.80659576 +0000 UTC m=+0.426544812,LastTimestamp:2026-09-04 23:36:26.80659576 +0000 UTC m=+0.426544812,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-66.ec2.internal,}"
Sep 04 23:36:26.826677 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.826603    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-66.ec2.internal.18d24261184e61c9\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-66.ec2.internal.18d24261184e61c9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-66.ec2.internal,UID:ip-10-0-138-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-66.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-66.ec2.internal,},FirstTimestamp:2026-09-04 23:36:26.806559177 +0000 UTC m=+0.426508233,LastTimestamp:2026-09-04 23:36:26.807851553 +0000 UTC m=+0.427800620,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-66.ec2.internal,}"
Sep 04 23:36:26.828695 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.828634    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-66.ec2.internal.18d24261184ec04d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-66.ec2.internal.18d24261184ec04d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-66.ec2.internal,UID:ip-10-0-138-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-66.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-66.ec2.internal,},FirstTimestamp:2026-09-04 23:36:26.806583373 +0000 UTC m=+0.426532421,LastTimestamp:2026-09-04 23:36:26.80787986 +0000 UTC m=+0.427828913,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-66.ec2.internal,}"
Sep 04 23:36:26.837510 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.837448    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-66.ec2.internal.18d24261184ef0b0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-66.ec2.internal.18d24261184ef0b0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-66.ec2.internal,UID:ip-10-0-138-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-66.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-66.ec2.internal,},FirstTimestamp:2026-09-04 23:36:26.80659576 +0000 UTC m=+0.426544812,LastTimestamp:2026-09-04 23:36:26.807895879 +0000 UTC m=+0.427844935,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-66.ec2.internal,}"
Sep 04 23:36:26.886752 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.886721    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-66.ec2.internal\" not found"
Sep 04 23:36:26.955701 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.955598    2562 policy_none.go:49] "None policy: Start"
Sep 04 23:36:26.955701 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.955652    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 04 23:36:26.955701 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.955670    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 04 23:36:26.957719 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.957699    2562 policy_none.go:47] "Start"
Sep 04 23:36:26.981057 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.976848    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 04 23:36:26.981057 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.978123    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 04 23:36:26.981057 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.978164    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 04 23:36:26.981057 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.978187    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 04 23:36:26.981057 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.978224    2562 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 04 23:36:26.981057 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.980179    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 04 23:36:26.982908 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.982894    2562 manager.go:337] "Starting Device Plugin manager"
Sep 04 23:36:26.982989 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.982935    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 04 23:36:26.982989 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.982949    2562 server.go:87] "Starting device plugin registration server"
Sep 04 23:36:26.983281 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.983267    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 04 23:36:26.983347 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.983281    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 04 23:36:26.983415 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.983396    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 04 23:36:26.983493 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.983476    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 04 23:36:26.983493 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:26.983492    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 04 23:36:26.983933 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.983917    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 04 23:36:26.983996 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.983958    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-66.ec2.internal\" not found"
Sep 04 23:36:26.999110 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:26.999089    2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-66.ec2.internal\" not found" node="ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.079281 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.079248    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-66.ec2.internal"]
Sep 04 23:36:27.079451 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.079335    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 23:36:27.080350 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.080335    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 23:36:27.080425 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.080364    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 23:36:27.080425 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.080385    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasSufficientPID"
Sep 04 23:36:27.082572 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.082559    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 23:36:27.082695 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.082681    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.082749 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.082712    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 23:36:27.083334 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.083315    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 23:36:27.083334 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.083334    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 23:36:27.083465 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.083345    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 23:36:27.083465 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.083356    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasSufficientPID"
Sep 04 23:36:27.083465 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.083318    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 23:36:27.083465 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.083411    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 23:36:27.083465 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.083423    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasSufficientPID"
Sep 04 23:36:27.084054 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.084040    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 23:36:27.084107 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.084064    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 23:36:27.084107 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.084075    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasSufficientPID"
Sep 04 23:36:27.084107 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.084099    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.085523 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.085507    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.085601 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.085531    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 23:36:27.086151 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.086136    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 23:36:27.086218 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.086163    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 23:36:27.086218 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.086173    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeHasSufficientPID"
Sep 04 23:36:27.088254 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.088232    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3e33ee9104d3ef59259d6c034d4b59a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal\" (UID: \"a3e33ee9104d3ef59259d6c034d4b59a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.088332 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.088270    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3e33ee9104d3ef59259d6c034d4b59a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal\" (UID: \"a3e33ee9104d3ef59259d6c034d4b59a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.088332 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.088309    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/34cc8fecf92fb93bf9c3003de27a7737-config\") pod \"kube-apiserver-proxy-ip-10-0-138-66.ec2.internal\" (UID: \"34cc8fecf92fb93bf9c3003de27a7737\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.090452 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.090437    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.090522 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:27.090457    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-66.ec2.internal\": node \"ip-10-0-138-66.ec2.internal\" not found"
Sep 04 23:36:27.106023 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:27.106007    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-66.ec2.internal\" not found" node="ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.110484 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:27.110467    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-66.ec2.internal\" not found" node="ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.113210 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:27.113196    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-66.ec2.internal\" not found"
Sep 04 23:36:27.188779 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.188747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3e33ee9104d3ef59259d6c034d4b59a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal\" (UID: \"a3e33ee9104d3ef59259d6c034d4b59a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.188779 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.188779    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3e33ee9104d3ef59259d6c034d4b59a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal\" (UID: \"a3e33ee9104d3ef59259d6c034d4b59a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.188950 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.188796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/34cc8fecf92fb93bf9c3003de27a7737-config\") pod \"kube-apiserver-proxy-ip-10-0-138-66.ec2.internal\" (UID: \"34cc8fecf92fb93bf9c3003de27a7737\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.188950 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.188866    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3e33ee9104d3ef59259d6c034d4b59a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal\" (UID: \"a3e33ee9104d3ef59259d6c034d4b59a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.188950 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.188875    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3e33ee9104d3ef59259d6c034d4b59a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal\" (UID: \"a3e33ee9104d3ef59259d6c034d4b59a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.188950 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.188917    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/34cc8fecf92fb93bf9c3003de27a7737-config\") pod \"kube-apiserver-proxy-ip-10-0-138-66.ec2.internal\" (UID: \"34cc8fecf92fb93bf9c3003de27a7737\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.213876 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:27.213824    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-66.ec2.internal\" not found"
Sep 04 23:36:27.314560 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:27.314519    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-66.ec2.internal\" not found"
Sep 04 23:36:27.409037 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.409002    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.415089 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:27.415052    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-66.ec2.internal\" not found"
Sep 04 23:36:27.415163 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:27.415099    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-66.ec2.internal"
Sep 04 23:36:27.515235 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:27.515155    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-66.ec2.internal\" not found"
Sep 04 23:36:28.988908 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:28.988886    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-66.ec2.internal\" not found"
Sep 04 23:36:29.012223 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:29.012203    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 23:36:29.089684 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:29.089639    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-66.ec2.internal\" not found"
Sep 04 23:36:29.095641 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:29.095622    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 04 23:36:29.095813 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:29.095791    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 04 23:36:29.095851 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:29.095818    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 23:36:29.190396 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:29.190328    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-66.ec2.internal\" not found"
Sep 04 23:36:29.290912 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:29.290868    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-66.ec2.internal\" not found"
Sep 04 23:36:29.306982 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:29.306963    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 23:36:29.330603 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:29.330569    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-03 23:31:28 +0000 UTC" deadline="2028-03-22 14:31:26.311917144 +0000 UTC"
Sep 04 23:36:29.330603 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:29.330599    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13550h54m56.981320081s"
Sep 04 23:36:29.360820 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:29.360793    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal"
Sep 04 23:36:29.373147 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:29.373122    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 04 23:36:29.374232 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:29.374220    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-66.ec2.internal"
Sep 04 23:36:29.382073 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:29.382054    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 04 23:36:29.554218 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:29.554061    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e33ee9104d3ef59259d6c034d4b59a.slice/crio-6ca5b2b392145140da497ed3c42a1afde7c5e246b74c75233deecc6f91a4089b WatchSource:0}: Error finding container 6ca5b2b392145140da497ed3c42a1afde7c5e246b74c75233deecc6f91a4089b: Status 404 returned error can't find the container with id 6ca5b2b392145140da497ed3c42a1afde7c5e246b74c75233deecc6f91a4089b
Sep 04 23:36:29.554744 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:29.554723    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34cc8fecf92fb93bf9c3003de27a7737.slice/crio-aa8449fbfe098d802c3312f27c15c5bfb83ddd2dc62cfa4114b90fd62341e215 WatchSource:0}: Error finding container aa8449fbfe098d802c3312f27c15c5bfb83ddd2dc62cfa4114b90fd62341e215: Status 404 returned error can't find the container with id aa8449fbfe098d802c3312f27c15c5bfb83ddd2dc62cfa4114b90fd62341e215
Sep 04 23:36:29.559728 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:29.559708    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 23:36:29.624215 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:29.624184    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 23:36:30.134394 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.134360    2562 apiserver.go:52] "Watching apiserver"
Sep 04 23:36:30.141369 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.141344    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 04 23:36:30.142663 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.142630    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-nc25g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx","openshift-image-registry/node-ca-pjgpk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal","openshift-multus/multus-additional-cni-plugins-47qkp","openshift-multus/multus-x2gpd","openshift-multus/network-metrics-daemon-m8mxs","openshift-network-diagnostics/network-check-target-mgg4d","kube-system/global-pull-secret-syncer-hq8l5","kube-system/kube-apiserver-proxy-ip-10-0-138-66.ec2.internal","openshift-cluster-node-tuning-operator/tuned-r4bwt","openshift-dns/node-resolver-dpw8x","openshift-network-operator/iptables-alerter-w7vt6","openshift-ovn-kubernetes/ovnkube-node-d75ld"]
Sep 04 23:36:30.145017 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.144995    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w7vt6"
Sep 04 23:36:30.147166 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.147144    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.147600 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.147569    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 04 23:36:30.147600 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.147590    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2fk7l\""
Sep 04 23:36:30.147762 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.147752    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 04 23:36:30.147816 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.147768    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 04 23:36:30.149263 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.149232    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 04 23:36:30.149491 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.149473    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 04 23:36:30.149552 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.149520    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 04 23:36:30.150209 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.149854    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pjgpk"
Sep 04 23:36:30.150209 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.149867    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bfhzv\""
Sep 04 23:36:30.151835 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.151817    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 04 23:36:30.152975 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.152401    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nc25g"
Sep 04 23:36:30.152975 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.152610    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 04 23:36:30.152975 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.152715    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-76xgw\""
Sep 04 23:36:30.154519 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.153484    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 04 23:36:30.155427 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.154749    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.155427 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.154796    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 04 23:36:30.155427 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.155003    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8ktjf\""
Sep 04 23:36:30.155427 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.155339    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 04 23:36:30.156821 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.156802    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 04 23:36:30.157080 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.157060    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.157182 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.157166    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 04 23:36:30.157361 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.157346    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 04 23:36:30.157432 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.157417    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 04 23:36:30.157539 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.157527    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 04 23:36:30.157589 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.157574    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mh9l8\""
Sep 04 23:36:30.159029 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.159013    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 04 23:36:30.159277 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.159261    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:30.159355 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.159336    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:30.159413 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.159378    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zsntb\""
Sep 04 23:36:30.160228 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.160210    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 04 23:36:30.161408 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.161391    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:30.161593 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.161570    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:30.163532 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.163515    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:30.163636 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.163579    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:30.165939 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.165921    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.168396 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.168239    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 04 23:36:30.168396 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.168329    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 04 23:36:30.168556 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.168430    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m4rh4\""
Sep 04 23:36:30.168556 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.168540    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 04 23:36:30.168686 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.168668    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.169520 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.169499    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 04 23:36:30.169520 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.169508    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 04 23:36:30.169723 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.169522    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 04 23:36:30.170258 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170135    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-etc-kubernetes\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.170258 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170169    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kdk6\" (UniqueName: \"kubernetes.io/projected/47ad65d3-8c19-41b5-b945-ed9944a35e94-kube-api-access-4kdk6\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.170258 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170199    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrhtq\" (UniqueName: \"kubernetes.io/projected/0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9-kube-api-access-qrhtq\") pod \"node-ca-pjgpk\" (UID: \"0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9\") " pod="openshift-image-registry/node-ca-pjgpk"
Sep 04 23:36:30.170258 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170222    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-socket-dir\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.170534 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170265    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-registration-dir\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.170534 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170303    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-run-k8s-cni-cncf-io\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.170534 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170364    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-var-lib-kubelet\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.170534 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48d8ba50-68de-4777-9e62-b7022698a5ca-agent-certs\") pod \"konnectivity-agent-nc25g\" (UID: \"48d8ba50-68de-4777-9e62-b7022698a5ca\") " pod="kube-system/konnectivity-agent-nc25g"
Sep 04 23:36:30.170534 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170417    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6a381b2a-9ed8-4b48-9079-20669cccde76-iptables-alerter-script\") pod \"iptables-alerter-w7vt6\" (UID: \"6a381b2a-9ed8-4b48-9079-20669cccde76\") " pod="openshift-network-operator/iptables-alerter-w7vt6"
Sep 04 23:36:30.170534 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-device-dir\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.170534 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170466    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-cnibin\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.170534 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170493    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48d8ba50-68de-4777-9e62-b7022698a5ca-konnectivity-ca\") pod \"konnectivity-agent-nc25g\" (UID: \"48d8ba50-68de-4777-9e62-b7022698a5ca\") " pod="kube-system/konnectivity-agent-nc25g"
Sep 04 23:36:30.170534 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs\") pod \"network-metrics-daemon-m8mxs\" (UID: \"38a67465-053c-4563-8c2c-8bff74b88a4b\") " pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170552    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-multus-cni-dir\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170581    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-multus-conf-dir\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170605    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47ad65d3-8c19-41b5-b945-ed9944a35e94-multus-daemon-config\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9-serviceca\") pod \"node-ca-pjgpk\" (UID: \"0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9\") " pod="openshift-image-registry/node-ca-pjgpk"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170647    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170689    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/133ae0cb-71c7-469e-bb5e-be538e8de7b3-cnibin\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170718    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75k4g\" (UniqueName: \"kubernetes.io/projected/133ae0cb-71c7-469e-bb5e-be538e8de7b3-kube-api-access-75k4g\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170744    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcbsv\" (UniqueName: \"kubernetes.io/projected/38a67465-053c-4563-8c2c-8bff74b88a4b-kube-api-access-tcbsv\") pod \"network-metrics-daemon-m8mxs\" (UID: \"38a67465-053c-4563-8c2c-8bff74b88a4b\") " pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170767    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-sys-fs\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-var-lib-cni-bin\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170812    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-var-lib-cni-multus\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170862    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9-host\") pod \"node-ca-pjgpk\" (UID: \"0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9\") " pod="openshift-image-registry/node-ca-pjgpk"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170900    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/133ae0cb-71c7-469e-bb5e-be538e8de7b3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170933    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a381b2a-9ed8-4b48-9079-20669cccde76-host-slash\") pod \"iptables-alerter-w7vt6\" (UID: \"6a381b2a-9ed8-4b48-9079-20669cccde76\") " pod="openshift-network-operator/iptables-alerter-w7vt6"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.170961    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-etc-selinux\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-multus-socket-dir-parent\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.171153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-run-multus-certs\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.172003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171049    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/133ae0cb-71c7-469e-bb5e-be538e8de7b3-cni-binary-copy\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.172003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171066    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/133ae0cb-71c7-469e-bb5e-be538e8de7b3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.172003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171106    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mb8t\" (UniqueName: \"kubernetes.io/projected/6a381b2a-9ed8-4b48-9079-20669cccde76-kube-api-access-2mb8t\") pod \"iptables-alerter-w7vt6\" (UID: \"6a381b2a-9ed8-4b48-9079-20669cccde76\") " pod="openshift-network-operator/iptables-alerter-w7vt6"
Sep 04 23:36:30.172003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171114    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x74bp\""
Sep 04 23:36:30.172003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171130    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.172003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171169    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngg8n\" (UniqueName: \"kubernetes.io/projected/0ae0632e-65ba-44e2-a612-62030cdc1fe2-kube-api-access-ngg8n\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.172003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171183    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-system-cni-dir\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.172003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171203    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47ad65d3-8c19-41b5-b945-ed9944a35e94-cni-binary-copy\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.172003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171216    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-run-netns\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.172003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/133ae0cb-71c7-469e-bb5e-be538e8de7b3-system-cni-dir\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.172003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171245    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/133ae0cb-71c7-469e-bb5e-be538e8de7b3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.172003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171285    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqmxf\" (UniqueName: \"kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf\") pod \"network-check-target-mgg4d\" (UID: \"20761599-3f21-4ba3-9a89-aceb760088a0\") " pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:30.172003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171320    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-hostroot\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.172003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171343    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 04 23:36:30.172003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.171749    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 04 23:36:30.172676 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.172231    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dpw8x"
Sep 04 23:36:30.174372 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.174353    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qjh8x\""
Sep 04 23:36:30.174372 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.174368    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 04 23:36:30.174505 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.174356    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 04 23:36:30.190879 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.190859    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-czn5v"
Sep 04 23:36:30.197764 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.197746    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-czn5v"
Sep 04 23:36:30.262024 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.261995    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 04 23:36:30.272078 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272044    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-registration-dir\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.272222 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272083    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-run-k8s-cni-cncf-io\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.272222 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-var-lib-kubelet\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.272222 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272128    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:30.272222 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272153    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-systemd\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.272222 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48d8ba50-68de-4777-9e62-b7022698a5ca-agent-certs\") pod \"konnectivity-agent-nc25g\" (UID: \"48d8ba50-68de-4777-9e62-b7022698a5ca\") " pod="kube-system/konnectivity-agent-nc25g"
Sep 04 23:36:30.272222 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272176    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-registration-dir\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.272499 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272236    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-run-k8s-cni-cncf-io\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.272499 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272188    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6a381b2a-9ed8-4b48-9079-20669cccde76-iptables-alerter-script\") pod \"iptables-alerter-w7vt6\" (UID: \"6a381b2a-9ed8-4b48-9079-20669cccde76\") " pod="openshift-network-operator/iptables-alerter-w7vt6"
Sep 04 23:36:30.272499 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-var-lib-kubelet\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.272499 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272309    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-ovn-node-metrics-cert\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.272499 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272357    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7234a20-6651-4386-8521-484e928babba-dbus\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:30.272499 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272393    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-var-lib-openvswitch\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.272499 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-log-socket\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.272499 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48d8ba50-68de-4777-9e62-b7022698a5ca-konnectivity-ca\") pod \"konnectivity-agent-nc25g\" (UID: \"48d8ba50-68de-4777-9e62-b7022698a5ca\") " pod="kube-system/konnectivity-agent-nc25g"
Sep 04 23:36:30.272499 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272485    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs\") pod \"network-metrics-daemon-m8mxs\" (UID: \"38a67465-053c-4563-8c2c-8bff74b88a4b\") " pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:30.272877 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.272572    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:36:30.272877 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.272630    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs podName:38a67465-053c-4563-8c2c-8bff74b88a4b nodeName:}" failed. No retries permitted until 2026-09-04 23:36:30.77261157 +0000 UTC m=+3.019278650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs") pod "network-metrics-daemon-m8mxs" (UID: "38a67465-053c-4563-8c2c-8bff74b88a4b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:36:30.272877 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272758    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6a381b2a-9ed8-4b48-9079-20669cccde76-iptables-alerter-script\") pod \"iptables-alerter-w7vt6\" (UID: \"6a381b2a-9ed8-4b48-9079-20669cccde76\") " pod="openshift-network-operator/iptables-alerter-w7vt6"
Sep 04 23:36:30.272877 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272811    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-multus-cni-dir\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.272877 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47ad65d3-8c19-41b5-b945-ed9944a35e94-multus-daemon-config\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.272877 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272857    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-run-ovn\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.273118 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272879    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-sysconfig\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.273118 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272900    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-kubernetes\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.273118 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272921    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pntnv\" (UniqueName: \"kubernetes.io/projected/cb4be02b-f45d-45f1-a512-2f7b3513a185-kube-api-access-pntnv\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.273118 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272945    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/133ae0cb-71c7-469e-bb5e-be538e8de7b3-cnibin\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.273118 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272951    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-multus-cni-dir\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.273118 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48d8ba50-68de-4777-9e62-b7022698a5ca-konnectivity-ca\") pod \"konnectivity-agent-nc25g\" (UID: \"48d8ba50-68de-4777-9e62-b7022698a5ca\") " pod="kube-system/konnectivity-agent-nc25g"
Sep 04 23:36:30.273118 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-var-lib-cni-multus\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.273118 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272994    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/133ae0cb-71c7-469e-bb5e-be538e8de7b3-cnibin\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.273118 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.272982    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 04 23:36:30.273118 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273009    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-run\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.273118 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273028    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-var-lib-cni-multus\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.273118 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f507b59a-26e6-4c41-b0ea-5a428a8adba8-tmp-dir\") pod \"node-resolver-dpw8x\" (UID: \"f507b59a-26e6-4c41-b0ea-5a428a8adba8\") " pod="openshift-dns/node-resolver-dpw8x"
Sep 04 23:36:30.273118 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273097    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9-host\") pod \"node-ca-pjgpk\" (UID: \"0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9\") " pod="openshift-image-registry/node-ca-pjgpk"
Sep 04 23:36:30.273118 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/133ae0cb-71c7-469e-bb5e-be538e8de7b3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273149    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9-host\") pod \"node-ca-pjgpk\" (UID: \"0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9\") " pod="openshift-image-registry/node-ca-pjgpk"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273144    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a381b2a-9ed8-4b48-9079-20669cccde76-host-slash\") pod \"iptables-alerter-w7vt6\" (UID: \"6a381b2a-9ed8-4b48-9079-20669cccde76\") " pod="openshift-network-operator/iptables-alerter-w7vt6"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273176    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-multus-socket-dir-parent\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273226    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a381b2a-9ed8-4b48-9079-20669cccde76-host-slash\") pod \"iptables-alerter-w7vt6\" (UID: \"6a381b2a-9ed8-4b48-9079-20669cccde76\") " pod="openshift-network-operator/iptables-alerter-w7vt6"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273243    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-multus-socket-dir-parent\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273265    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-systemd-units\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273289    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-env-overrides\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273316    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/133ae0cb-71c7-469e-bb5e-be538e8de7b3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273343    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngg8n\" (UniqueName: \"kubernetes.io/projected/0ae0632e-65ba-44e2-a612-62030cdc1fe2-kube-api-access-ngg8n\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273367    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47ad65d3-8c19-41b5-b945-ed9944a35e94-cni-binary-copy\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273390    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-run-netns\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-cni-netd\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/133ae0cb-71c7-469e-bb5e-be538e8de7b3-system-cni-dir\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/133ae0cb-71c7-469e-bb5e-be538e8de7b3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-ovnkube-config\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273504    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqrlc\" (UniqueName: \"kubernetes.io/projected/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-kube-api-access-gqrlc\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.273849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273528    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb4be02b-f45d-45f1-a512-2f7b3513a185-tmp\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47ad65d3-8c19-41b5-b945-ed9944a35e94-multus-daemon-config\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273552    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrhtq\" (UniqueName: \"kubernetes.io/projected/0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9-kube-api-access-qrhtq\") pod \"node-ca-pjgpk\" (UID: \"0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9\") " pod="openshift-image-registry/node-ca-pjgpk"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-run-systemd\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273611    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-sysctl-conf\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/133ae0cb-71c7-469e-bb5e-be538e8de7b3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273636    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-var-lib-kubelet\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273698    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-device-dir\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273715    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-run-netns\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273724    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-cnibin\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273769    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-kubelet\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-run-openvswitch\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273814    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-tuned\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273843    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/133ae0cb-71c7-469e-bb5e-be538e8de7b3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273851    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-node-log\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273613    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/133ae0cb-71c7-469e-bb5e-be538e8de7b3-system-cni-dir\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273858    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-cnibin\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.274670 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273873    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-ovnkube-script-lib\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273879    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/133ae0cb-71c7-469e-bb5e-be538e8de7b3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273902    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-multus-conf-dir\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-device-dir\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273933    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-sysctl-d\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273952    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-multus-conf-dir\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273956    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-lib-modules\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.273985    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f507b59a-26e6-4c41-b0ea-5a428a8adba8-hosts-file\") pod \"node-resolver-dpw8x\" (UID: \"f507b59a-26e6-4c41-b0ea-5a428a8adba8\") " pod="openshift-dns/node-resolver-dpw8x"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274011    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9-serviceca\") pod \"node-ca-pjgpk\" (UID: \"0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9\") " pod="openshift-image-registry/node-ca-pjgpk"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274027    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47ad65d3-8c19-41b5-b945-ed9944a35e94-cni-binary-copy\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274038    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75k4g\" (UniqueName: \"kubernetes.io/projected/133ae0cb-71c7-469e-bb5e-be538e8de7b3-kube-api-access-75k4g\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274067    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcbsv\" (UniqueName: \"kubernetes.io/projected/38a67465-053c-4563-8c2c-8bff74b88a4b-kube-api-access-tcbsv\") pod \"network-metrics-daemon-m8mxs\" (UID: \"38a67465-053c-4563-8c2c-8bff74b88a4b\") " pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-sys-fs\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274136    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-var-lib-cni-bin\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-sys-fs\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274167    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-run-netns\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274192    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.275612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274215    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-cni-bin\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-etc-selinux\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-run-multus-certs\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274335    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-run-multus-certs\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274337    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9-serviceca\") pod \"node-ca-pjgpk\" (UID: \"0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9\") " pod="openshift-image-registry/node-ca-pjgpk"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274370    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-host-var-lib-cni-bin\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274375    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7234a20-6651-4386-8521-484e928babba-kubelet-config\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274402    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-etc-selinux\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274418    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-modprobe-d\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274442    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/133ae0cb-71c7-469e-bb5e-be538e8de7b3-cni-binary-copy\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274465    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mb8t\" (UniqueName: \"kubernetes.io/projected/6a381b2a-9ed8-4b48-9079-20669cccde76-kube-api-access-2mb8t\") pod \"iptables-alerter-w7vt6\" (UID: \"6a381b2a-9ed8-4b48-9079-20669cccde76\") " pod="openshift-network-operator/iptables-alerter-w7vt6"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274559    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-system-cni-dir\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-hostroot\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274607    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-etc-kubernetes\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274632    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-slash\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.276473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274687    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-hostroot\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.277313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274687    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-etc-openvswitch\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.277313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274725    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-system-cni-dir\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.277313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274736    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47ad65d3-8c19-41b5-b945-ed9944a35e94-etc-kubernetes\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.277313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274746    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmxf\" (UniqueName: \"kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf\") pod \"network-check-target-mgg4d\" (UID: \"20761599-3f21-4ba3-9a89-aceb760088a0\") " pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:30.277313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274778    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kdk6\" (UniqueName: \"kubernetes.io/projected/47ad65d3-8c19-41b5-b945-ed9944a35e94-kube-api-access-4kdk6\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.277313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274922    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/133ae0cb-71c7-469e-bb5e-be538e8de7b3-cni-binary-copy\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.277313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274936    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.277313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274963    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-sys\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.277313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.274990    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-host\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.277313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.275006    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9szp\" (UniqueName: \"kubernetes.io/projected/f507b59a-26e6-4c41-b0ea-5a428a8adba8-kube-api-access-f9szp\") pod \"node-resolver-dpw8x\" (UID: \"f507b59a-26e6-4c41-b0ea-5a428a8adba8\") " pod="openshift-dns/node-resolver-dpw8x"
Sep 04 23:36:30.277313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.275042    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-socket-dir\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.277313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.275152    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0632e-65ba-44e2-a612-62030cdc1fe2-socket-dir\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.277898 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.277513    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48d8ba50-68de-4777-9e62-b7022698a5ca-agent-certs\") pod \"konnectivity-agent-nc25g\" (UID: \"48d8ba50-68de-4777-9e62-b7022698a5ca\") " pod="kube-system/konnectivity-agent-nc25g"
Sep 04 23:36:30.284210 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.284188    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 23:36:30.284330 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.284213    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 23:36:30.284330 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.284228    2562 projected.go:196] Error preparing data for projected volume kube-api-access-kqmxf for pod openshift-network-diagnostics/network-check-target-mgg4d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:36:30.284330 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.284284    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf podName:20761599-3f21-4ba3-9a89-aceb760088a0 nodeName:}" failed. No retries permitted until 2026-09-04 23:36:30.784267781 +0000 UTC m=+3.030934859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kqmxf" (UniqueName: "kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf") pod "network-check-target-mgg4d" (UID: "20761599-3f21-4ba3-9a89-aceb760088a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:36:30.285333 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.285307    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrhtq\" (UniqueName: \"kubernetes.io/projected/0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9-kube-api-access-qrhtq\") pod \"node-ca-pjgpk\" (UID: \"0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9\") " pod="openshift-image-registry/node-ca-pjgpk"
Sep 04 23:36:30.286166 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.286148    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kdk6\" (UniqueName: \"kubernetes.io/projected/47ad65d3-8c19-41b5-b945-ed9944a35e94-kube-api-access-4kdk6\") pod \"multus-x2gpd\" (UID: \"47ad65d3-8c19-41b5-b945-ed9944a35e94\") " pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.286683 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.286638    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngg8n\" (UniqueName: \"kubernetes.io/projected/0ae0632e-65ba-44e2-a612-62030cdc1fe2-kube-api-access-ngg8n\") pod \"aws-ebs-csi-driver-node-j8tvx\" (UID: \"0ae0632e-65ba-44e2-a612-62030cdc1fe2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.287551 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.287508    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcbsv\" (UniqueName: \"kubernetes.io/projected/38a67465-053c-4563-8c2c-8bff74b88a4b-kube-api-access-tcbsv\") pod \"network-metrics-daemon-m8mxs\" (UID: \"38a67465-053c-4563-8c2c-8bff74b88a4b\") " pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:30.287551 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.287523    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mb8t\" (UniqueName: \"kubernetes.io/projected/6a381b2a-9ed8-4b48-9079-20669cccde76-kube-api-access-2mb8t\") pod \"iptables-alerter-w7vt6\" (UID: \"6a381b2a-9ed8-4b48-9079-20669cccde76\") " pod="openshift-network-operator/iptables-alerter-w7vt6"
Sep 04 23:36:30.288001 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.287942    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75k4g\" (UniqueName: \"kubernetes.io/projected/133ae0cb-71c7-469e-bb5e-be538e8de7b3-kube-api-access-75k4g\") pod \"multus-additional-cni-plugins-47qkp\" (UID: \"133ae0cb-71c7-469e-bb5e-be538e8de7b3\") " pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.356622 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.356564    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-66.ec2.internal" event={"ID":"34cc8fecf92fb93bf9c3003de27a7737","Type":"ContainerStarted","Data":"aa8449fbfe098d802c3312f27c15c5bfb83ddd2dc62cfa4114b90fd62341e215"}
Sep 04 23:36:30.358003 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.357964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal" event={"ID":"a3e33ee9104d3ef59259d6c034d4b59a","Type":"ContainerStarted","Data":"6ca5b2b392145140da497ed3c42a1afde7c5e246b74c75233deecc6f91a4089b"}
Sep 04 23:36:30.375551 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375520    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-node-log\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.375705 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375564    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-ovnkube-script-lib\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.375705 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-sysctl-d\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.375705 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375612    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-lib-modules\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.375705 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375635    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f507b59a-26e6-4c41-b0ea-5a428a8adba8-hosts-file\") pod \"node-resolver-dpw8x\" (UID: \"f507b59a-26e6-4c41-b0ea-5a428a8adba8\") " pod="openshift-dns/node-resolver-dpw8x"
Sep 04 23:36:30.375705 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375642    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-node-log\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.375705 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-run-netns\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.375705 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375702    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.375979 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375724    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-cni-bin\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.375979 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7234a20-6651-4386-8521-484e928babba-kubelet-config\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:30.375979 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375774    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-modprobe-d\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.375979 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375799    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-lib-modules\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.375979 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375802    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-slash\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.375979 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375842    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-etc-openvswitch\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.375979 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.375979 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375899    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f507b59a-26e6-4c41-b0ea-5a428a8adba8-hosts-file\") pod \"node-resolver-dpw8x\" (UID: \"f507b59a-26e6-4c41-b0ea-5a428a8adba8\") " pod="openshift-dns/node-resolver-dpw8x"
Sep 04 23:36:30.375979 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375909    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-sys\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.375979 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375933    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-host\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.375979 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375959    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9szp\" (UniqueName: \"kubernetes.io/projected/f507b59a-26e6-4c41-b0ea-5a428a8adba8-kube-api-access-f9szp\") pod \"node-resolver-dpw8x\" (UID: \"f507b59a-26e6-4c41-b0ea-5a428a8adba8\") " pod="openshift-dns/node-resolver-dpw8x"
Sep 04 23:36:30.375979 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375995    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376020    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-systemd\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-ovn-node-metrics-cert\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376074    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7234a20-6651-4386-8521-484e928babba-dbus\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376103    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-var-lib-openvswitch\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376127    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-log-socket\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376166    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-run-ovn\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376188    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-sysconfig\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376224    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-ovnkube-script-lib\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375845    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-slash\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376290    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-etc-openvswitch\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376312    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-sys\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376331    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376333    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-sysctl-d\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376229    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-kubernetes\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376360    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-host\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.376498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376368    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-kubernetes\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntnv\" (UniqueName: \"kubernetes.io/projected/cb4be02b-f45d-45f1-a512-2f7b3513a185-kube-api-access-pntnv\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376399    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-cni-bin\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376403    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-run\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376439    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f507b59a-26e6-4c41-b0ea-5a428a8adba8-tmp-dir\") pod \"node-resolver-dpw8x\" (UID: \"f507b59a-26e6-4c41-b0ea-5a428a8adba8\") " pod="openshift-dns/node-resolver-dpw8x"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376461    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-modprobe-d\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376470    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-systemd-units\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376497    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-env-overrides\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376502    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-run\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376516    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-var-lib-openvswitch\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-log-socket\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376571    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-cni-netd\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376598    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-ovnkube-config\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376603    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7234a20-6651-4386-8521-484e928babba-kubelet-config\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.376613    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376621    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqrlc\" (UniqueName: \"kubernetes.io/projected/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-kube-api-access-gqrlc\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376644    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb4be02b-f45d-45f1-a512-2f7b3513a185-tmp\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.377282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376685    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-systemd-units\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.376690    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret podName:e7234a20-6651-4386-8521-484e928babba nodeName:}" failed. No retries permitted until 2026-09-04 23:36:30.876674249 +0000 UTC m=+3.123341320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret") pod "global-pull-secret-syncer-hq8l5" (UID: "e7234a20-6651-4386-8521-484e928babba") : object "kube-system"/"original-pull-secret" not registered
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376726    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-run-systemd\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376753    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-sysctl-conf\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-var-lib-kubelet\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376804    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-kubelet\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-run-openvswitch\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376851    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-tuned\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376878    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-systemd\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.375936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-run-netns\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376923    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-env-overrides\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-run-systemd\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376964    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-sysconfig\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.376983    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-kubelet\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.377001    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-run-ovn\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.377013    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-run-openvswitch\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.377025    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-var-lib-kubelet\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.378114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.377059    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-host-cni-netd\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.378915 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.377089    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-sysctl-conf\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.378915 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.377140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7234a20-6651-4386-8521-484e928babba-dbus\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:30.378915 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.377560    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-ovnkube-config\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.379321 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.379290    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb4be02b-f45d-45f1-a512-2f7b3513a185-tmp\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.379406 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.379372    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb4be02b-f45d-45f1-a512-2f7b3513a185-etc-tuned\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.381337 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.381314    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-ovn-node-metrics-cert\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.384768 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.384719    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pntnv\" (UniqueName: \"kubernetes.io/projected/cb4be02b-f45d-45f1-a512-2f7b3513a185-kube-api-access-pntnv\") pod \"tuned-r4bwt\" (UID: \"cb4be02b-f45d-45f1-a512-2f7b3513a185\") " pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.385645 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.385625    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqrlc\" (UniqueName: \"kubernetes.io/projected/3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee-kube-api-access-gqrlc\") pod \"ovnkube-node-d75ld\" (UID: \"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.386838 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.386814    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f507b59a-26e6-4c41-b0ea-5a428a8adba8-tmp-dir\") pod \"node-resolver-dpw8x\" (UID: \"f507b59a-26e6-4c41-b0ea-5a428a8adba8\") " pod="openshift-dns/node-resolver-dpw8x"
Sep 04 23:36:30.387083 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.387056    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9szp\" (UniqueName: \"kubernetes.io/projected/f507b59a-26e6-4c41-b0ea-5a428a8adba8-kube-api-access-f9szp\") pod \"node-resolver-dpw8x\" (UID: \"f507b59a-26e6-4c41-b0ea-5a428a8adba8\") " pod="openshift-dns/node-resolver-dpw8x"
Sep 04 23:36:30.459613 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.459510    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w7vt6"
Sep 04 23:36:30.467225 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.467006    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx"
Sep 04 23:36:30.471569 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:30.471534    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a381b2a_9ed8_4b48_9079_20669cccde76.slice/crio-39b71daa5e05a3a4d1f7e80a16432374dfbf12c3bff6ad514e069b32c065763c WatchSource:0}: Error finding container 39b71daa5e05a3a4d1f7e80a16432374dfbf12c3bff6ad514e069b32c065763c: Status 404 returned error can't find the container with id 39b71daa5e05a3a4d1f7e80a16432374dfbf12c3bff6ad514e069b32c065763c
Sep 04 23:36:30.477813 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.477581    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pjgpk"
Sep 04 23:36:30.481596 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.481358    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nc25g"
Sep 04 23:36:30.487633 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.487431    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-47qkp"
Sep 04 23:36:30.492666 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:30.492626    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d8ba50_68de_4777_9e62_b7022698a5ca.slice/crio-b73ab6a4bff82134e2cd154931c79f6ad0a396b8c449df074e3d89c88555bee2 WatchSource:0}: Error finding container b73ab6a4bff82134e2cd154931c79f6ad0a396b8c449df074e3d89c88555bee2: Status 404 returned error can't find the container with id b73ab6a4bff82134e2cd154931c79f6ad0a396b8c449df074e3d89c88555bee2
Sep 04 23:36:30.494069 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.494047    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x2gpd"
Sep 04 23:36:30.494331 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:30.494306    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ffbbbd9_0ce0_4b3c_9a58_964ebeadf9a9.slice/crio-465dc42edb753a50383377d5f0224ea1495bf58742cf158a5f3d055732025ddf WatchSource:0}: Error finding container 465dc42edb753a50383377d5f0224ea1495bf58742cf158a5f3d055732025ddf: Status 404 returned error can't find the container with id 465dc42edb753a50383377d5f0224ea1495bf58742cf158a5f3d055732025ddf
Sep 04 23:36:30.500777 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.500685    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:30.503289 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:30.503248    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133ae0cb_71c7_469e_bb5e_be538e8de7b3.slice/crio-5c3701b8aa69e691ffa61efb840983113ca087c5a8a7a47debc9d1f8465efd00 WatchSource:0}: Error finding container 5c3701b8aa69e691ffa61efb840983113ca087c5a8a7a47debc9d1f8465efd00: Status 404 returned error can't find the container with id 5c3701b8aa69e691ffa61efb840983113ca087c5a8a7a47debc9d1f8465efd00
Sep 04 23:36:30.506406 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:30.506360    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ad65d3_8c19_41b5_b945_ed9944a35e94.slice/crio-5b1fffc3826afc6530cc1b62f5c6ec3ac9b9de9bfddf3f90258684c7ad6fc03c WatchSource:0}: Error finding container 5b1fffc3826afc6530cc1b62f5c6ec3ac9b9de9bfddf3f90258684c7ad6fc03c: Status 404 returned error can't find the container with id 5b1fffc3826afc6530cc1b62f5c6ec3ac9b9de9bfddf3f90258684c7ad6fc03c
Sep 04 23:36:30.507912 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.507537    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r4bwt"
Sep 04 23:36:30.512790 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:30.512767    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e04aab2_c27c_4ba1_b1df_3d2daa4f13ee.slice/crio-ffc7dae25c8e399a785b8b0234b456eb54e58ef5ce771dc48b72031b84d3d424 WatchSource:0}: Error finding container ffc7dae25c8e399a785b8b0234b456eb54e58ef5ce771dc48b72031b84d3d424: Status 404 returned error can't find the container with id ffc7dae25c8e399a785b8b0234b456eb54e58ef5ce771dc48b72031b84d3d424
Sep 04 23:36:30.512885 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.512829    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dpw8x"
Sep 04 23:36:30.519128 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:36:30.519104    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4be02b_f45d_45f1_a512_2f7b3513a185.slice/crio-5dbc4dae7fde84055e6edd5d47ab34b749cae2e54151df223ddc8d63802526b6 WatchSource:0}: Error finding container 5dbc4dae7fde84055e6edd5d47ab34b749cae2e54151df223ddc8d63802526b6: Status 404 returned error can't find the container with id 5dbc4dae7fde84055e6edd5d47ab34b749cae2e54151df223ddc8d63802526b6
Sep 04 23:36:30.779849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.779747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs\") pod \"network-metrics-daemon-m8mxs\" (UID: \"38a67465-053c-4563-8c2c-8bff74b88a4b\") " pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:30.780008 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.779905    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:36:30.780008 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.779974    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs podName:38a67465-053c-4563-8c2c-8bff74b88a4b nodeName:}" failed. No retries permitted until 2026-09-04 23:36:31.779954227 +0000 UTC m=+4.026621284 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs") pod "network-metrics-daemon-m8mxs" (UID: "38a67465-053c-4563-8c2c-8bff74b88a4b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:36:30.880798 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.880762    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmxf\" (UniqueName: \"kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf\") pod \"network-check-target-mgg4d\" (UID: \"20761599-3f21-4ba3-9a89-aceb760088a0\") " pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:30.880981 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.880818    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:30.880981 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.880942    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 23:36:30.880981 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.880981    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 23:36:30.881119 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.880995    2562 projected.go:196] Error preparing data for projected volume kube-api-access-kqmxf for pod openshift-network-diagnostics/network-check-target-mgg4d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:36:30.881119 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.880945    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 23:36:30.881119 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.881046    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf podName:20761599-3f21-4ba3-9a89-aceb760088a0 nodeName:}" failed. No retries permitted until 2026-09-04 23:36:31.881029617 +0000 UTC m=+4.127696690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kqmxf" (UniqueName: "kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf") pod "network-check-target-mgg4d" (UID: "20761599-3f21-4ba3-9a89-aceb760088a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:36:30.881119 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:30.881082    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret podName:e7234a20-6651-4386-8521-484e928babba nodeName:}" failed. No retries permitted until 2026-09-04 23:36:31.881069134 +0000 UTC m=+4.127736190 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret") pod "global-pull-secret-syncer-hq8l5" (UID: "e7234a20-6651-4386-8521-484e928babba") : object "kube-system"/"original-pull-secret" not registered
Sep 04 23:36:30.976219 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:30.976182    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 23:36:31.200901 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.200790    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-03 23:31:30 +0000 UTC" deadline="2028-06-03 10:07:10.566420432 +0000 UTC"
Sep 04 23:36:31.200901 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.200845    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15298h30m39.365580369s"
Sep 04 23:36:31.351928 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.351846    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:31.352096 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.351950    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:31.352096 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:31.351959    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:31.352096 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.351984    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:31.352096 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:31.352086    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:31.352291 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:31.352165    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:31.360994 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.360963    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld" event={"ID":"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee","Type":"ContainerStarted","Data":"ffc7dae25c8e399a785b8b0234b456eb54e58ef5ce771dc48b72031b84d3d424"}
Sep 04 23:36:31.362589 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.362564    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x2gpd" event={"ID":"47ad65d3-8c19-41b5-b945-ed9944a35e94","Type":"ContainerStarted","Data":"5b1fffc3826afc6530cc1b62f5c6ec3ac9b9de9bfddf3f90258684c7ad6fc03c"}
Sep 04 23:36:31.364119 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.364078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47qkp" event={"ID":"133ae0cb-71c7-469e-bb5e-be538e8de7b3","Type":"ContainerStarted","Data":"5c3701b8aa69e691ffa61efb840983113ca087c5a8a7a47debc9d1f8465efd00"}
Sep 04 23:36:31.365573 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.365549    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nc25g" event={"ID":"48d8ba50-68de-4777-9e62-b7022698a5ca","Type":"ContainerStarted","Data":"b73ab6a4bff82134e2cd154931c79f6ad0a396b8c449df074e3d89c88555bee2"}
Sep 04 23:36:31.367034 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.367009    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pjgpk" event={"ID":"0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9","Type":"ContainerStarted","Data":"465dc42edb753a50383377d5f0224ea1495bf58742cf158a5f3d055732025ddf"}
Sep 04 23:36:31.368277 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.368240    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx" event={"ID":"0ae0632e-65ba-44e2-a612-62030cdc1fe2","Type":"ContainerStarted","Data":"166d558f02849c3ac3a1a9f839a31fc056ba4f9f03851c5f3b984833efe91ead"}
Sep 04 23:36:31.369479 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.369460    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r4bwt" event={"ID":"cb4be02b-f45d-45f1-a512-2f7b3513a185","Type":"ContainerStarted","Data":"5dbc4dae7fde84055e6edd5d47ab34b749cae2e54151df223ddc8d63802526b6"}
Sep 04 23:36:31.371233 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.371208    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w7vt6" event={"ID":"6a381b2a-9ed8-4b48-9079-20669cccde76","Type":"ContainerStarted","Data":"39b71daa5e05a3a4d1f7e80a16432374dfbf12c3bff6ad514e069b32c065763c"}
Sep 04 23:36:31.788261 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.787688    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs\") pod \"network-metrics-daemon-m8mxs\" (UID: \"38a67465-053c-4563-8c2c-8bff74b88a4b\") " pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:31.788261 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:31.787844    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:36:31.788261 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:31.787911    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs podName:38a67465-053c-4563-8c2c-8bff74b88a4b nodeName:}" failed. No retries permitted until 2026-09-04 23:36:33.787890124 +0000 UTC m=+6.034557200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs") pod "network-metrics-daemon-m8mxs" (UID: "38a67465-053c-4563-8c2c-8bff74b88a4b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:36:31.888496 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.888453    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmxf\" (UniqueName: \"kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf\") pod \"network-check-target-mgg4d\" (UID: \"20761599-3f21-4ba3-9a89-aceb760088a0\") " pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:31.888694 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:31.888590    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:31.888827 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:31.888808    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 23:36:31.888922 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:31.888875    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret podName:e7234a20-6651-4386-8521-484e928babba nodeName:}" failed. No retries permitted until 2026-09-04 23:36:33.888856966 +0000 UTC m=+6.135524046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret") pod "global-pull-secret-syncer-hq8l5" (UID: "e7234a20-6651-4386-8521-484e928babba") : object "kube-system"/"original-pull-secret" not registered
Sep 04 23:36:31.889339 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:31.889319    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 23:36:31.889437 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:31.889343    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 23:36:31.889437 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:31.889356    2562 projected.go:196] Error preparing data for projected volume kube-api-access-kqmxf for pod openshift-network-diagnostics/network-check-target-mgg4d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:36:31.889437 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:31.889404    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf podName:20761599-3f21-4ba3-9a89-aceb760088a0 nodeName:}" failed. No retries permitted until 2026-09-04 23:36:33.889388194 +0000 UTC m=+6.136055256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kqmxf" (UniqueName: "kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf") pod "network-check-target-mgg4d" (UID: "20761599-3f21-4ba3-9a89-aceb760088a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:36:32.201090 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:32.201059    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-03 23:31:30 +0000 UTC" deadline="2028-03-19 06:30:57.255505921 +0000 UTC"
Sep 04 23:36:32.201447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:32.201091    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13470h54m25.05441935s"
Sep 04 23:36:32.381396 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:32.381317    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dpw8x" event={"ID":"f507b59a-26e6-4c41-b0ea-5a428a8adba8","Type":"ContainerStarted","Data":"c5985c3e3f8e926ed610b61e58077b2c54168b32ec6cf057a90910bddd9950a0"}
Sep 04 23:36:32.392414 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:32.392332    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-66.ec2.internal" event={"ID":"34cc8fecf92fb93bf9c3003de27a7737","Type":"ContainerStarted","Data":"97ff209d7312db3970cc8ebddae028dd58b8ecc5e6d71d66cda1df60fb3a6eef"}
Sep 04 23:36:32.406294 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:32.406237    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-66.ec2.internal" podStartSLOduration=3.406219328 podStartE2EDuration="3.406219328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:36:32.405967121 +0000 UTC m=+4.652634201" watchObservedRunningTime="2026-09-04 23:36:32.406219328 +0000 UTC m=+4.652886408"
Sep 04 23:36:33.352584 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:33.352552    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:33.353318 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:33.352734    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:33.353318 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:33.353103    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:33.353318 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:33.353199    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:33.353318 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:33.353250    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:33.353318 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:33.353319    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:33.405583 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:33.405537    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal" event={"ID":"a3e33ee9104d3ef59259d6c034d4b59a","Type":"ContainerStarted","Data":"361c8280d33f45d359c18deb91007cef7e41479025fafc1fa6da43f27bf5b06d"}
Sep 04 23:36:33.802918 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:33.802801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs\") pod \"network-metrics-daemon-m8mxs\" (UID: \"38a67465-053c-4563-8c2c-8bff74b88a4b\") " pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:33.803095 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:33.802960    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:36:33.803095 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:33.803026    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs podName:38a67465-053c-4563-8c2c-8bff74b88a4b nodeName:}" failed. No retries permitted until 2026-09-04 23:36:37.803005808 +0000 UTC m=+10.049672866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs") pod "network-metrics-daemon-m8mxs" (UID: "38a67465-053c-4563-8c2c-8bff74b88a4b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:36:33.904067 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:33.904028    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmxf\" (UniqueName: \"kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf\") pod \"network-check-target-mgg4d\" (UID: \"20761599-3f21-4ba3-9a89-aceb760088a0\") " pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:33.904232 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:33.904088    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:33.904289 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:33.904247    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 23:36:33.904339 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:33.904306    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret podName:e7234a20-6651-4386-8521-484e928babba nodeName:}" failed. No retries permitted until 2026-09-04 23:36:37.904289094 +0000 UTC m=+10.150956173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret") pod "global-pull-secret-syncer-hq8l5" (UID: "e7234a20-6651-4386-8521-484e928babba") : object "kube-system"/"original-pull-secret" not registered
Sep 04 23:36:33.904724 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:33.904704    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 23:36:33.904839 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:33.904730    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 23:36:33.904839 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:33.904742    2562 projected.go:196] Error preparing data for projected volume kube-api-access-kqmxf for pod openshift-network-diagnostics/network-check-target-mgg4d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:36:33.904839 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:33.904783    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf podName:20761599-3f21-4ba3-9a89-aceb760088a0 nodeName:}" failed. No retries permitted until 2026-09-04 23:36:37.904770323 +0000 UTC m=+10.151437380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kqmxf" (UniqueName: "kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf") pod "network-check-target-mgg4d" (UID: "20761599-3f21-4ba3-9a89-aceb760088a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:36:35.352398 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:35.352136    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:35.352398 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:35.352276    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:35.353106 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:35.352835    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:35.353106 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:35.352883    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:35.353106 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:35.352959    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:35.353106 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:35.353050    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:35.411648 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:35.411432    2562 generic.go:356] "Generic (PLEG): container finished" podID="a3e33ee9104d3ef59259d6c034d4b59a" containerID="361c8280d33f45d359c18deb91007cef7e41479025fafc1fa6da43f27bf5b06d" exitCode=0
Sep 04 23:36:35.411648 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:35.411483    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal" event={"ID":"a3e33ee9104d3ef59259d6c034d4b59a","Type":"ContainerDied","Data":"361c8280d33f45d359c18deb91007cef7e41479025fafc1fa6da43f27bf5b06d"}
Sep 04 23:36:37.352633 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:37.352598    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:37.353111 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:37.352615    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:37.353111 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:37.352757    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:37.353111 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:37.352770    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:37.353111 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:37.352883    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:37.353111 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:37.352956    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:37.838224 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:37.838072    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs\") pod \"network-metrics-daemon-m8mxs\" (UID: \"38a67465-053c-4563-8c2c-8bff74b88a4b\") " pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:37.838381 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:37.838246    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:36:37.838381 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:37.838329    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs podName:38a67465-053c-4563-8c2c-8bff74b88a4b nodeName:}" failed. No retries permitted until 2026-09-04 23:36:45.83830793 +0000 UTC m=+18.084975001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs") pod "network-metrics-daemon-m8mxs" (UID: "38a67465-053c-4563-8c2c-8bff74b88a4b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:36:37.939502 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:37.938940    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmxf\" (UniqueName: \"kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf\") pod \"network-check-target-mgg4d\" (UID: \"20761599-3f21-4ba3-9a89-aceb760088a0\") " pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:37.939502 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:37.938995    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:37.939502 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:37.939125    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 23:36:37.939502 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:37.939154    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 23:36:37.939502 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:37.939177    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 23:36:37.939502 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:37.939186    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret podName:e7234a20-6651-4386-8521-484e928babba nodeName:}" failed. No retries permitted until 2026-09-04 23:36:45.93916814 +0000 UTC m=+18.185835218 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret") pod "global-pull-secret-syncer-hq8l5" (UID: "e7234a20-6651-4386-8521-484e928babba") : object "kube-system"/"original-pull-secret" not registered
Sep 04 23:36:37.939502 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:37.939192    2562 projected.go:196] Error preparing data for projected volume kube-api-access-kqmxf for pod openshift-network-diagnostics/network-check-target-mgg4d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:36:37.939502 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:37.939247    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf podName:20761599-3f21-4ba3-9a89-aceb760088a0 nodeName:}" failed. No retries permitted until 2026-09-04 23:36:45.939230494 +0000 UTC m=+18.185897564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kqmxf" (UniqueName: "kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf") pod "network-check-target-mgg4d" (UID: "20761599-3f21-4ba3-9a89-aceb760088a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:36:39.352245 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:39.351993    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:39.352767 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:39.352204    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:39.352767 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:39.352339    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:39.352767 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:39.352204    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:39.352767 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:39.352436    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:39.352767 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:39.352499    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:41.352308 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:41.352276    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:41.352738 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:41.352276    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:41.352738 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:41.352385    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:41.352738 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:41.352275    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:41.352738 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:41.352451    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:41.352738 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:41.352549    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:43.352116 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:43.352078    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:43.352116 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:43.352108    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:43.352576 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:43.352077    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:43.352576 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:43.352220    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:43.352576 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:43.352296    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:43.352576 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:43.352372    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:45.352554 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:45.352519    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:45.352554 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:45.352548    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:45.353094 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:45.352558    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:45.353094 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:45.352678    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:45.353094 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:45.352747    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:45.353094 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:45.352815    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:45.895133 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:45.895099    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs\") pod \"network-metrics-daemon-m8mxs\" (UID: \"38a67465-053c-4563-8c2c-8bff74b88a4b\") " pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:45.895273 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:45.895225    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:36:45.895329 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:45.895289    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs podName:38a67465-053c-4563-8c2c-8bff74b88a4b nodeName:}" failed. No retries permitted until 2026-09-04 23:37:01.895274145 +0000 UTC m=+34.141941201 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs") pod "network-metrics-daemon-m8mxs" (UID: "38a67465-053c-4563-8c2c-8bff74b88a4b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:36:45.996206 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:45.996167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmxf\" (UniqueName: \"kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf\") pod \"network-check-target-mgg4d\" (UID: \"20761599-3f21-4ba3-9a89-aceb760088a0\") " pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:45.996350 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:45.996316    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:45.996350 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:45.996335    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 23:36:45.996450 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:45.996361    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 23:36:45.996450 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:45.996374    2562 projected.go:196] Error preparing data for projected volume kube-api-access-kqmxf for pod openshift-network-diagnostics/network-check-target-mgg4d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:36:45.996450 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:45.996440    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf podName:20761599-3f21-4ba3-9a89-aceb760088a0 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:01.996420513 +0000 UTC m=+34.243087579 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kqmxf" (UniqueName: "kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf") pod "network-check-target-mgg4d" (UID: "20761599-3f21-4ba3-9a89-aceb760088a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:36:45.996612 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:45.996448    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 23:36:45.996612 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:45.996505    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret podName:e7234a20-6651-4386-8521-484e928babba nodeName:}" failed. No retries permitted until 2026-09-04 23:37:01.996488738 +0000 UTC m=+34.243155813 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret") pod "global-pull-secret-syncer-hq8l5" (UID: "e7234a20-6651-4386-8521-484e928babba") : object "kube-system"/"original-pull-secret" not registered
Sep 04 23:36:47.352523 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:47.352484    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:47.353018 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:47.352484    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:47.353018 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:47.352600    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:47.353018 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:47.352711    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:47.353018 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:47.352489    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:47.353018 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:47.352790    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:49.351954 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.351665    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:49.352541 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.351722    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:49.352541 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:49.352023    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:49.352541 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.351760    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:49.352541 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:49.352164    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:49.352541 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:49.352097    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:49.439152 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.439069    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal" event={"ID":"a3e33ee9104d3ef59259d6c034d4b59a","Type":"ContainerStarted","Data":"ec276867b2c389a53edff664a284f8ddd060a35e3ade7bc61e3f3201bfc7dcdd"}
Sep 04 23:36:49.441810 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.441789    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 04 23:36:49.442139 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.442117    2562 generic.go:356] "Generic (PLEG): container finished" podID="3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee" containerID="27b1f56065a9f591024a7d6757e9abb20f92b731a3e7c5d123c6253d82074276" exitCode=1
Sep 04 23:36:49.442222 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.442196    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld" event={"ID":"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee","Type":"ContainerStarted","Data":"991248e44ae0616e6e2f2ab41123fc0c6e64b787abb96f79f3ea47fe7141ddbd"}
Sep 04 23:36:49.442294 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.442279    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld" event={"ID":"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee","Type":"ContainerStarted","Data":"d6b0da78554ff370ab0f9abcf6604289c8975ca057ec9dc9578b4e450c26e76e"}
Sep 04 23:36:49.442353 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.442336    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld" event={"ID":"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee","Type":"ContainerStarted","Data":"ee04c7473eca7f13c6a38ed35022dedab60de2bed449f8d1592da92fe333d183"}
Sep 04 23:36:49.442402 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.442388    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld" event={"ID":"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee","Type":"ContainerStarted","Data":"dc9226983eafe75f3b07a2afa9c07dcae6a8d3fa436e7ba57261ae874fa70d2f"}
Sep 04 23:36:49.442450 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.442437    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld" event={"ID":"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee","Type":"ContainerDied","Data":"27b1f56065a9f591024a7d6757e9abb20f92b731a3e7c5d123c6253d82074276"}
Sep 04 23:36:49.442516 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.442489    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld" event={"ID":"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee","Type":"ContainerStarted","Data":"fe396562400544a3e7c7e02163f36377c6d1fd28cdc93b1be4efcab36ab33051"}
Sep 04 23:36:49.443611 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.443588    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x2gpd" event={"ID":"47ad65d3-8c19-41b5-b945-ed9944a35e94","Type":"ContainerStarted","Data":"6c2165fbb02beb1511a04cf489d945773806af1a06c466629f5a9131df7f5f0c"}
Sep 04 23:36:49.445225 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.445200    2562 generic.go:356] "Generic (PLEG): container finished" podID="133ae0cb-71c7-469e-bb5e-be538e8de7b3" containerID="f24459b54cbd16f3e90db95a018ab7cbc5351c93274b8e4e012584419d2e7476" exitCode=0
Sep 04 23:36:49.445331 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.445286    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47qkp" event={"ID":"133ae0cb-71c7-469e-bb5e-be538e8de7b3","Type":"ContainerDied","Data":"f24459b54cbd16f3e90db95a018ab7cbc5351c93274b8e4e012584419d2e7476"}
Sep 04 23:36:49.447149 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.447101    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nc25g" event={"ID":"48d8ba50-68de-4777-9e62-b7022698a5ca","Type":"ContainerStarted","Data":"55a3e44f6df9fe7738ab5a5c7573015f95cf69fba5d93100bc80854a81071bcb"}
Sep 04 23:36:49.449267 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.449240    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pjgpk" event={"ID":"0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9","Type":"ContainerStarted","Data":"9abf748c53175f471eb13ad50e7c4f8d2f5975440cd51a904fbcf4692134c2e2"}
Sep 04 23:36:49.450956 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.450931    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx" event={"ID":"0ae0632e-65ba-44e2-a612-62030cdc1fe2","Type":"ContainerStarted","Data":"32b5349296e7af5e28b12cb973f2defef0d8b4112f870a68b75c1b88e4d9e68f"}
Sep 04 23:36:49.452577 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.452557    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r4bwt" event={"ID":"cb4be02b-f45d-45f1-a512-2f7b3513a185","Type":"ContainerStarted","Data":"f6b31917a06af1d56b6ca8ae425d05664041507175253e612ffc71ea67c8e06e"}
Sep 04 23:36:49.453621 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.453584    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-66.ec2.internal" podStartSLOduration=20.453569945 podStartE2EDuration="20.453569945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:36:49.453421849 +0000 UTC m=+21.700088932" watchObservedRunningTime="2026-09-04 23:36:49.453569945 +0000 UTC m=+21.700237027"
Sep 04 23:36:49.454098 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.454080    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dpw8x" event={"ID":"f507b59a-26e6-4c41-b0ea-5a428a8adba8","Type":"ContainerStarted","Data":"1213885d45492e5444b449b9e291737c30ccb98eaf9a53d03fece3283e103609"}
Sep 04 23:36:49.465873 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.465830    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pjgpk" podStartSLOduration=3.428068928 podStartE2EDuration="21.465814086s" totalImagesPullingTime="18.037745158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:36:49.465691141 +0000 UTC m=+21.712358220" watchObservedRunningTime="2026-09-04 23:36:49.465814086 +0000 UTC m=+21.712481166"
Sep 04 23:36:49.489777 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.489740    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-x2gpd" podStartSLOduration=3.218385718 podStartE2EDuration="21.489727696s" totalImagesPullingTime="18.271341978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:36:49.489541133 +0000 UTC m=+21.736208212" watchObservedRunningTime="2026-09-04 23:36:49.489727696 +0000 UTC m=+21.736394775"
Sep 04 23:36:49.502768 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.502710    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nc25g" podStartSLOduration=3.469656399 podStartE2EDuration="21.502691995s" totalImagesPullingTime="18.033035596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:36:49.502387607 +0000 UTC m=+21.749054686" watchObservedRunningTime="2026-09-04 23:36:49.502691995 +0000 UTC m=+21.749359074"
Sep 04 23:36:49.537908 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.537860    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r4bwt" podStartSLOduration=3.517089306 podStartE2EDuration="21.537843289s" totalImagesPullingTime="18.020753983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:36:49.537743249 +0000 UTC m=+21.784410329" watchObservedRunningTime="2026-09-04 23:36:49.537843289 +0000 UTC m=+21.784510369"
Sep 04 23:36:49.551377 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.551329    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dpw8x" podStartSLOduration=5.206990824 podStartE2EDuration="21.551312976s" totalImagesPullingTime="16.344322152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:36:49.551086878 +0000 UTC m=+21.797753958" watchObservedRunningTime="2026-09-04 23:36:49.551312976 +0000 UTC m=+21.797980096"
Sep 04 23:36:49.935861 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:49.935831    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 04 23:36:50.370019 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:50.369892    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-04T23:36:49.935859753Z","UUID":"2cbdbd87-9c69-45d6-8818-e6972216ba25","Handler":null,"Name":"","Endpoint":""}
Sep 04 23:36:50.371204 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:50.371184    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 04 23:36:50.371329 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:50.371217    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 04 23:36:50.458463 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:50.458430    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx" event={"ID":"0ae0632e-65ba-44e2-a612-62030cdc1fe2","Type":"ContainerStarted","Data":"c5b39b9ac54831f1bf135fc1f1be5297203a7df290fd78c8d320e45a411f0016"}
Sep 04 23:36:50.460183 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:50.460153    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w7vt6" event={"ID":"6a381b2a-9ed8-4b48-9079-20669cccde76","Type":"ContainerStarted","Data":"fc34045b88995e6523258b1a196b54fa2cbcc7d9274308fa304fd0a09eee3f3f"}
Sep 04 23:36:50.473794 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:50.473738    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w7vt6" podStartSLOduration=4.410458229 podStartE2EDuration="22.473719451s" totalImagesPullingTime="18.063261222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:36:50.47353592 +0000 UTC m=+22.720203002" watchObservedRunningTime="2026-09-04 23:36:50.473719451 +0000 UTC m=+22.720386530"
Sep 04 23:36:51.352596 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:51.352560    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:51.352596 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:51.352591    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:51.352813 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:51.352705    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:51.352813 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:51.352725    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:51.352813 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:51.352788    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:51.352961 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:51.352881    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:51.465949 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:51.465920    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 04 23:36:51.466333 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:51.466306    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld" event={"ID":"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee","Type":"ContainerStarted","Data":"e23839c1873a6f23e39f8219a4c68f88ec37135b47e1f548134dc11e1369a4d2"}
Sep 04 23:36:51.468180 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:51.468158    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx" event={"ID":"0ae0632e-65ba-44e2-a612-62030cdc1fe2","Type":"ContainerStarted","Data":"fb42bad50e4382829642b2aec9e71ff1f6bcd40a1f20482bab4a57d7bdbd4f7b"}
Sep 04 23:36:51.484314 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:51.484275    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8tvx" podStartSLOduration=3.379544166 podStartE2EDuration="23.484262233s" totalImagesPullingTime="20.104718067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:36:51.483755725 +0000 UTC m=+23.730422804" watchObservedRunningTime="2026-09-04 23:36:51.484262233 +0000 UTC m=+23.730929313"
Sep 04 23:36:52.813280 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:52.813083    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nc25g"
Sep 04 23:36:52.813805 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:52.813786    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nc25g"
Sep 04 23:36:53.352628 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:53.352597    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:53.352784 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:53.352602    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:53.352784 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:53.352725    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:53.352784 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:53.352773    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:53.352937 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:53.352889    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:53.352987 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:53.352958    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:54.480628 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:54.480573    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 04 23:36:54.481512 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:54.481478    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld" event={"ID":"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee","Type":"ContainerStarted","Data":"c5f44685db8226aa65ef997e0c1001c8f9f736981d9dccf3ed352e6f853e7a40"}
Sep 04 23:36:54.481801 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:54.481732    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:54.481991 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:54.481970    2562 scope.go:117] "RemoveContainer" containerID="27b1f56065a9f591024a7d6757e9abb20f92b731a3e7c5d123c6253d82074276"
Sep 04 23:36:54.486900 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:54.486861    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47qkp" event={"ID":"133ae0cb-71c7-469e-bb5e-be538e8de7b3","Type":"ContainerStarted","Data":"21a512c846f34220c99b0774e943edfc3b91fc8866a690c5a5652430411cc91a"}
Sep 04 23:36:54.501746 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:54.501718    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:55.352176 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:55.352140    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:55.352374 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:55.352153    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:55.352374 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:55.352293    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:55.352374 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:55.352304    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:55.352541 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:55.352402    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:55.352541 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:55.352487    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:55.493088 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:55.493056    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 04 23:36:55.493553 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:55.493457    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld" event={"ID":"3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee","Type":"ContainerStarted","Data":"38aded3fe0f19750eb61a1f0f3ff7c1dfa0c291bd891f1d52e1e8ee1b47fef0c"}
Sep 04 23:36:55.493772 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:55.493750    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:55.493875 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:55.493819    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:55.495377 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:55.495349    2562 generic.go:356] "Generic (PLEG): container finished" podID="133ae0cb-71c7-469e-bb5e-be538e8de7b3" containerID="21a512c846f34220c99b0774e943edfc3b91fc8866a690c5a5652430411cc91a" exitCode=0
Sep 04 23:36:55.495491 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:55.495405    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47qkp" event={"ID":"133ae0cb-71c7-469e-bb5e-be538e8de7b3","Type":"ContainerDied","Data":"21a512c846f34220c99b0774e943edfc3b91fc8866a690c5a5652430411cc91a"}
Sep 04 23:36:55.508553 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:55.508533    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:36:55.518582 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:55.518548    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld" podStartSLOduration=9.435839826 podStartE2EDuration="27.518536004s" totalImagesPullingTime="18.082696178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:36:55.517396026 +0000 UTC m=+27.764063104" watchObservedRunningTime="2026-09-04 23:36:55.518536004 +0000 UTC m=+27.765203083"
Sep 04 23:36:56.461901 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:56.461692    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hq8l5"]
Sep 04 23:36:56.462166 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:56.461984    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:56.462166 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:56.462098    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:56.464588 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:56.464564    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m8mxs"]
Sep 04 23:36:56.464743 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:56.464672    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:56.464808 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:56.464789    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:56.475785 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:56.475763    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mgg4d"]
Sep 04 23:36:56.475871 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:56.475859    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:56.475942 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:56.475926    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:56.499710 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:56.499676    2562 generic.go:356] "Generic (PLEG): container finished" podID="133ae0cb-71c7-469e-bb5e-be538e8de7b3" containerID="d4246392ed68b6991245e1975b314986be84810cee6de28b8a56b399e73caa31" exitCode=0
Sep 04 23:36:56.500157 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:56.499758    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47qkp" event={"ID":"133ae0cb-71c7-469e-bb5e-be538e8de7b3","Type":"ContainerDied","Data":"d4246392ed68b6991245e1975b314986be84810cee6de28b8a56b399e73caa31"}
Sep 04 23:36:57.504950 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:57.504920    2562 generic.go:356] "Generic (PLEG): container finished" podID="133ae0cb-71c7-469e-bb5e-be538e8de7b3" containerID="2d599b3d970c34b67bff410a77a7e52f93f735f7f733d83a4e68d29af319895e" exitCode=0
Sep 04 23:36:57.505377 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:57.505005    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47qkp" event={"ID":"133ae0cb-71c7-469e-bb5e-be538e8de7b3","Type":"ContainerDied","Data":"2d599b3d970c34b67bff410a77a7e52f93f735f7f733d83a4e68d29af319895e"}
Sep 04 23:36:58.353972 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:58.353913    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:36:58.353972 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:58.353935    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:36:58.353972 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:58.353944    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:36:58.354211 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:58.354042    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:36:58.354211 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:58.354176    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:36:58.354313 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:36:58.354268    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:36:59.480312 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:59.480278    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nc25g"
Sep 04 23:36:59.480785 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:59.480451    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 04 23:36:59.480987 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:36:59.480969    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nc25g"
Sep 04 23:37:00.352283 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:00.352251    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:37:00.352469 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:00.352251    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:37:00.352469 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:00.352377    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8mxs" podUID="38a67465-053c-4563-8c2c-8bff74b88a4b"
Sep 04 23:37:00.352558 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:00.352461    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hq8l5" podUID="e7234a20-6651-4386-8521-484e928babba"
Sep 04 23:37:00.352558 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:00.352251    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:37:00.352644 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:00.352553    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mgg4d" podUID="20761599-3f21-4ba3-9a89-aceb760088a0"
Sep 04 23:37:01.526413 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.526343    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-66.ec2.internal" event="NodeReady"
Sep 04 23:37:01.526852 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.526487    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 04 23:37:01.559038 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.558997    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687"]
Sep 04 23:37:01.602383 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.602351    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cdf99854b-t9dhs"]
Sep 04 23:37:01.602543 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.602482    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687"
Sep 04 23:37:01.605083 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.605057    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 04 23:37:01.605211 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.605064    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 04 23:37:01.605270 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.605255    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wbbp8\""
Sep 04 23:37:01.605438 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.605416    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 04 23:37:01.605559 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.605542    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 04 23:37:01.615778 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.615759    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp"]
Sep 04 23:37:01.615910 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.615894    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.618071 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.618047    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 04 23:37:01.618161 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.618109    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 04 23:37:01.618161 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.618120    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6hx2z\""
Sep 04 23:37:01.618794 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.618775    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 04 23:37:01.624326 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.624305    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 04 23:37:01.633813 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.633790    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-5c7r9"]
Sep 04 23:37:01.633950 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.633918    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp"
Sep 04 23:37:01.637085 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.637065    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 04 23:37:01.637201 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.637108    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kj4xj\""
Sep 04 23:37:01.637367 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.637341    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 04 23:37:01.637455 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.637369    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 04 23:37:01.637455 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.637380    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 04 23:37:01.659909 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.659887    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-vr466"]
Sep 04 23:37:01.660048 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.660016    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:01.662880 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.662846    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 04 23:37:01.662880 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.662897    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8lcs8\""
Sep 04 23:37:01.663112 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.662856    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 04 23:37:01.663337 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.663319    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 04 23:37:01.663452 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.663364    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 04 23:37:01.668955 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.668938    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 04 23:37:01.675831 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.675810    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-rllx5"]
Sep 04 23:37:01.675940 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.675922    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-vr466"
Sep 04 23:37:01.678008 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.677974    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-gcvc5\""
Sep 04 23:37:01.678106 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.677975    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 04 23:37:01.678106 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.678077    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 04 23:37:01.689983 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.689965    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"]
Sep 04 23:37:01.692745 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.690463    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-rllx5"
Sep 04 23:37:01.693498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.693241    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 04 23:37:01.693498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.693308    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 04 23:37:01.693498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.693492    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xtrjq\""
Sep 04 23:37:01.707717 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.707694    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-installation-pull-secrets\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.707813 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.707730    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b6f2cdb-29a1-4245-83c3-1bf5eb182073-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bz687\" (UID: \"2b6f2cdb-29a1-4245-83c3-1bf5eb182073\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687"
Sep 04 23:37:01.707813 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.707763    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.707813 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.707790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-trusted-ca\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.707951 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.707813    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-bound-sa-token\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.707951 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.707838    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k86bx\" (UniqueName: \"kubernetes.io/projected/2b6f2cdb-29a1-4245-83c3-1bf5eb182073-kube-api-access-k86bx\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bz687\" (UID: \"2b6f2cdb-29a1-4245-83c3-1bf5eb182073\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687"
Sep 04 23:37:01.707951 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.707889    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-image-registry-private-configuration\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.707951 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.707913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4ll6\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-kube-api-access-w4ll6\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.707951 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.707939    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91cf8c41-30cb-4ce3-a9b1-193bdce73a60-config\") pod \"service-ca-operator-859f94c4b-fbpjp\" (UID: \"91cf8c41-30cb-4ce3-a9b1-193bdce73a60\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp"
Sep 04 23:37:01.708179 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.707960    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b6f2cdb-29a1-4245-83c3-1bf5eb182073-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bz687\" (UID: \"2b6f2cdb-29a1-4245-83c3-1bf5eb182073\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687"
Sep 04 23:37:01.708179 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.707987    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-certificates\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.708179 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.708011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91cf8c41-30cb-4ce3-a9b1-193bdce73a60-serving-cert\") pod \"service-ca-operator-859f94c4b-fbpjp\" (UID: \"91cf8c41-30cb-4ce3-a9b1-193bdce73a60\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp"
Sep 04 23:37:01.708179 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.708114    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-ca-trust-extracted\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.708179 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.708165    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5lhg\" (UniqueName: \"kubernetes.io/projected/91cf8c41-30cb-4ce3-a9b1-193bdce73a60-kube-api-access-c5lhg\") pod \"service-ca-operator-859f94c4b-fbpjp\" (UID: \"91cf8c41-30cb-4ce3-a9b1-193bdce73a60\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp"
Sep 04 23:37:01.710082 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.710062    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687"]
Sep 04 23:37:01.710154 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.710094    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fmc2z"]
Sep 04 23:37:01.710195 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.710162    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"
Sep 04 23:37:01.712309 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.712260    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 04 23:37:01.712404 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.712348    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 04 23:37:01.712534 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.712519    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2fcnr\""
Sep 04 23:37:01.712617 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.712576    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 04 23:37:01.732120 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.732092    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp"]
Sep 04 23:37:01.732253 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.732129    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-vr466"]
Sep 04 23:37:01.732253 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.732148    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-rllx5"]
Sep 04 23:37:01.732253 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.732165    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-5c7r9"]
Sep 04 23:37:01.732253 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.732182    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cdf99854b-t9dhs"]
Sep 04 23:37:01.732253 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.732205    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rv66q"]
Sep 04 23:37:01.732480 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.732358    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fmc2z"
Sep 04 23:37:01.734589 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.734562    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 04 23:37:01.734589 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.734603    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 04 23:37:01.734853 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.734717    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 04 23:37:01.734959 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.734923    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ttwxb\""
Sep 04 23:37:01.756204 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.756183    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fmc2z"]
Sep 04 23:37:01.756329 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.756211    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"]
Sep 04 23:37:01.756329 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.756230    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rv66q"]
Sep 04 23:37:01.756401 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.756331    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:01.758593 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.758573    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 04 23:37:01.758593 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.758581    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 04 23:37:01.758593 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.758574    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4rhln\""
Sep 04 23:37:01.809498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809424    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-installation-pull-secrets\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.809498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809458    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b6f2cdb-29a1-4245-83c3-1bf5eb182073-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bz687\" (UID: \"2b6f2cdb-29a1-4245-83c3-1bf5eb182073\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687"
Sep 04 23:37:01.809498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809477    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d903250-92e3-48c2-b52f-e70ff30805a0-cert\") pod \"ingress-canary-fmc2z\" (UID: \"3d903250-92e3-48c2-b52f-e70ff30805a0\") " pod="openshift-ingress-canary/ingress-canary-fmc2z"
Sep 04 23:37:01.809795 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809501    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmdmh\" (UniqueName: \"kubernetes.io/projected/9ad47c0d-7f00-4653-85fb-333ca37395d0-kube-api-access-pmdmh\") pod \"volume-data-source-validator-695c895bd6-vr466\" (UID: \"9ad47c0d-7f00-4653-85fb-333ca37395d0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-vr466"
Sep 04 23:37:01.809795 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809529    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.809795 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809554    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c3a8a40-d2d2-4093-85dc-25b176df91d2-config\") pod \"console-operator-85b454874b-5c7r9\" (UID: \"8c3a8a40-d2d2-4093-85dc-25b176df91d2\") " pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:01.809795 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809574    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-trusted-ca\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.809795 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-bound-sa-token\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.809795 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809607    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k86bx\" (UniqueName: \"kubernetes.io/projected/2b6f2cdb-29a1-4245-83c3-1bf5eb182073-kube-api-access-k86bx\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bz687\" (UID: \"2b6f2cdb-29a1-4245-83c3-1bf5eb182073\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687"
Sep 04 23:37:01.809795 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809635    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs5fg\" (UniqueName: \"kubernetes.io/projected/3d903250-92e3-48c2-b52f-e70ff30805a0-kube-api-access-bs5fg\") pod \"ingress-canary-fmc2z\" (UID: \"3d903250-92e3-48c2-b52f-e70ff30805a0\") " pod="openshift-ingress-canary/ingress-canary-fmc2z"
Sep 04 23:37:01.809795 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:01.809694    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 04 23:37:01.809795 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:01.809717    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cdf99854b-t9dhs: secret "image-registry-tls" not found
Sep 04 23:37:01.809795 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809711    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-image-registry-private-configuration\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.809795 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:01.809778    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls podName:ceeabe27-d2d7-4819-aa74-cc9609f8e3e6 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:02.309756529 +0000 UTC m=+34.556423608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls") pod "image-registry-cdf99854b-t9dhs" (UID: "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6") : secret "image-registry-tls" not found
Sep 04 23:37:01.810447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809817    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4ll6\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-kube-api-access-w4ll6\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.810447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809851    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91cf8c41-30cb-4ce3-a9b1-193bdce73a60-config\") pod \"service-ca-operator-859f94c4b-fbpjp\" (UID: \"91cf8c41-30cb-4ce3-a9b1-193bdce73a60\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp"
Sep 04 23:37:01.810447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809878    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b6f2cdb-29a1-4245-83c3-1bf5eb182073-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bz687\" (UID: \"2b6f2cdb-29a1-4245-83c3-1bf5eb182073\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687"
Sep 04 23:37:01.810447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809911    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c3a8a40-d2d2-4093-85dc-25b176df91d2-serving-cert\") pod \"console-operator-85b454874b-5c7r9\" (UID: \"8c3a8a40-d2d2-4093-85dc-25b176df91d2\") " pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:01.810447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809949    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-certificates\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.810447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809972    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91cf8c41-30cb-4ce3-a9b1-193bdce73a60-serving-cert\") pod \"service-ca-operator-859f94c4b-fbpjp\" (UID: \"91cf8c41-30cb-4ce3-a9b1-193bdce73a60\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp"
Sep 04 23:37:01.810447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.809999    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c3a8a40-d2d2-4093-85dc-25b176df91d2-trusted-ca\") pod \"console-operator-85b454874b-5c7r9\" (UID: \"8c3a8a40-d2d2-4093-85dc-25b176df91d2\") " pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:01.810447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.810024    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9r6\" (UniqueName: \"kubernetes.io/projected/8c3a8a40-d2d2-4093-85dc-25b176df91d2-kube-api-access-cq9r6\") pod \"console-operator-85b454874b-5c7r9\" (UID: \"8c3a8a40-d2d2-4093-85dc-25b176df91d2\") " pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:01.810447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.810140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b6f2cdb-29a1-4245-83c3-1bf5eb182073-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bz687\" (UID: \"2b6f2cdb-29a1-4245-83c3-1bf5eb182073\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687"
Sep 04 23:37:01.810447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.810201    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-ca-trust-extracted\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.810447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.810228    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wfn79\" (UID: \"51e702f1-195b-4e70-b454-ffe4228685ae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"
Sep 04 23:37:01.810447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.810279    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lhg\" (UniqueName: \"kubernetes.io/projected/91cf8c41-30cb-4ce3-a9b1-193bdce73a60-kube-api-access-c5lhg\") pod \"service-ca-operator-859f94c4b-fbpjp\" (UID: \"91cf8c41-30cb-4ce3-a9b1-193bdce73a60\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp"
Sep 04 23:37:01.810447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.810303    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p89nm\" (UniqueName: \"kubernetes.io/projected/81b45393-bdad-471b-b018-b0b51925dcfe-kube-api-access-p89nm\") pod \"network-check-source-685456655f-rllx5\" (UID: \"81b45393-bdad-471b-b018-b0b51925dcfe\") " pod="openshift-network-diagnostics/network-check-source-685456655f-rllx5"
Sep 04 23:37:01.810447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.810327    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvxdj\" (UniqueName: \"kubernetes.io/projected/51e702f1-195b-4e70-b454-ffe4228685ae-kube-api-access-wvxdj\") pod \"cluster-samples-operator-7cdcfcf76c-wfn79\" (UID: \"51e702f1-195b-4e70-b454-ffe4228685ae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"
Sep 04 23:37:01.810447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.810345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91cf8c41-30cb-4ce3-a9b1-193bdce73a60-config\") pod \"service-ca-operator-859f94c4b-fbpjp\" (UID: \"91cf8c41-30cb-4ce3-a9b1-193bdce73a60\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp"
Sep 04 23:37:01.813237 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.813210    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b6f2cdb-29a1-4245-83c3-1bf5eb182073-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bz687\" (UID: \"2b6f2cdb-29a1-4245-83c3-1bf5eb182073\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687"
Sep 04 23:37:01.813331 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.813212    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91cf8c41-30cb-4ce3-a9b1-193bdce73a60-serving-cert\") pod \"service-ca-operator-859f94c4b-fbpjp\" (UID: \"91cf8c41-30cb-4ce3-a9b1-193bdce73a60\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp"
Sep 04 23:37:01.819172 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.819128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5lhg\" (UniqueName: \"kubernetes.io/projected/91cf8c41-30cb-4ce3-a9b1-193bdce73a60-kube-api-access-c5lhg\") pod \"service-ca-operator-859f94c4b-fbpjp\" (UID: \"91cf8c41-30cb-4ce3-a9b1-193bdce73a60\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp"
Sep 04 23:37:01.819274 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.819215    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-certificates\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.819645 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.819618    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-ca-trust-extracted\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.819770 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.819694    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k86bx\" (UniqueName: \"kubernetes.io/projected/2b6f2cdb-29a1-4245-83c3-1bf5eb182073-kube-api-access-k86bx\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bz687\" (UID: \"2b6f2cdb-29a1-4245-83c3-1bf5eb182073\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687"
Sep 04 23:37:01.820321 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.820293    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-trusted-ca\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.820972 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.820952    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4ll6\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-kube-api-access-w4ll6\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.821052 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.820953    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-image-registry-private-configuration\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.821137 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.821121    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-installation-pull-secrets\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.821275 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.821254    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-bound-sa-token\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:01.911404 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.911365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p89nm\" (UniqueName: \"kubernetes.io/projected/81b45393-bdad-471b-b018-b0b51925dcfe-kube-api-access-p89nm\") pod \"network-check-source-685456655f-rllx5\" (UID: \"81b45393-bdad-471b-b018-b0b51925dcfe\") " pod="openshift-network-diagnostics/network-check-source-685456655f-rllx5"
Sep 04 23:37:01.911404 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.911406    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvxdj\" (UniqueName: \"kubernetes.io/projected/51e702f1-195b-4e70-b454-ffe4228685ae-kube-api-access-wvxdj\") pod \"cluster-samples-operator-7cdcfcf76c-wfn79\" (UID: \"51e702f1-195b-4e70-b454-ffe4228685ae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"
Sep 04 23:37:01.911643 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.911436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs\") pod \"network-metrics-daemon-m8mxs\" (UID: \"38a67465-053c-4563-8c2c-8bff74b88a4b\") " pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:37:01.911643 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.911460    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d903250-92e3-48c2-b52f-e70ff30805a0-cert\") pod \"ingress-canary-fmc2z\" (UID: \"3d903250-92e3-48c2-b52f-e70ff30805a0\") " pod="openshift-ingress-canary/ingress-canary-fmc2z"
Sep 04 23:37:01.911643 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.911478    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f745f9b6-37b0-4710-a39f-282b4d388677-config-volume\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:01.911643 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.911497    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmdmh\" (UniqueName: \"kubernetes.io/projected/9ad47c0d-7f00-4653-85fb-333ca37395d0-kube-api-access-pmdmh\") pod \"volume-data-source-validator-695c895bd6-vr466\" (UID: \"9ad47c0d-7f00-4653-85fb-333ca37395d0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-vr466"
Sep 04 23:37:01.911643 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.911511    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f745f9b6-37b0-4710-a39f-282b4d388677-tmp-dir\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:01.911643 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.911582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c3a8a40-d2d2-4093-85dc-25b176df91d2-config\") pod \"console-operator-85b454874b-5c7r9\" (UID: \"8c3a8a40-d2d2-4093-85dc-25b176df91d2\") " pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:01.911961 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:01.911706    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:01.911961 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.911713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs5fg\" (UniqueName: \"kubernetes.io/projected/3d903250-92e3-48c2-b52f-e70ff30805a0-kube-api-access-bs5fg\") pod \"ingress-canary-fmc2z\" (UID: \"3d903250-92e3-48c2-b52f-e70ff30805a0\") " pod="openshift-ingress-canary/ingress-canary-fmc2z"
Sep 04 23:37:01.911961 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.911748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:01.911961 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:01.911767    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs podName:38a67465-053c-4563-8c2c-8bff74b88a4b nodeName:}" failed. No retries permitted until 2026-09-04 23:37:33.911749298 +0000 UTC m=+66.158416368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs") pod "network-metrics-daemon-m8mxs" (UID: "38a67465-053c-4563-8c2c-8bff74b88a4b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:01.911961 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.911809    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmbhz\" (UniqueName: \"kubernetes.io/projected/f745f9b6-37b0-4710-a39f-282b4d388677-kube-api-access-kmbhz\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:01.911961 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.911842    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c3a8a40-d2d2-4093-85dc-25b176df91d2-serving-cert\") pod \"console-operator-85b454874b-5c7r9\" (UID: \"8c3a8a40-d2d2-4093-85dc-25b176df91d2\") " pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:01.911961 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.911884    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c3a8a40-d2d2-4093-85dc-25b176df91d2-trusted-ca\") pod \"console-operator-85b454874b-5c7r9\" (UID: \"8c3a8a40-d2d2-4093-85dc-25b176df91d2\") " pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:01.911961 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.911921    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9r6\" (UniqueName: \"kubernetes.io/projected/8c3a8a40-d2d2-4093-85dc-25b176df91d2-kube-api-access-cq9r6\") pod \"console-operator-85b454874b-5c7r9\" (UID: \"8c3a8a40-d2d2-4093-85dc-25b176df91d2\") " pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:01.912280 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.911969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wfn79\" (UID: \"51e702f1-195b-4e70-b454-ffe4228685ae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"
Sep 04 23:37:01.912280 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:01.912125    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 04 23:37:01.912280 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:01.912200    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls podName:51e702f1-195b-4e70-b454-ffe4228685ae nodeName:}" failed. No retries permitted until 2026-09-04 23:37:02.412179884 +0000 UTC m=+34.658846949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-wfn79" (UID: "51e702f1-195b-4e70-b454-ffe4228685ae") : secret "samples-operator-tls" not found
Sep 04 23:37:01.912385 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.912289    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c3a8a40-d2d2-4093-85dc-25b176df91d2-config\") pod \"console-operator-85b454874b-5c7r9\" (UID: \"8c3a8a40-d2d2-4093-85dc-25b176df91d2\") " pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:01.912735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.912719    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c3a8a40-d2d2-4093-85dc-25b176df91d2-trusted-ca\") pod \"console-operator-85b454874b-5c7r9\" (UID: \"8c3a8a40-d2d2-4093-85dc-25b176df91d2\") " pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:01.912802 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.912792    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687"
Sep 04 23:37:01.914293 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.914260    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c3a8a40-d2d2-4093-85dc-25b176df91d2-serving-cert\") pod \"console-operator-85b454874b-5c7r9\" (UID: \"8c3a8a40-d2d2-4093-85dc-25b176df91d2\") " pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:01.914398 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.914307    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d903250-92e3-48c2-b52f-e70ff30805a0-cert\") pod \"ingress-canary-fmc2z\" (UID: \"3d903250-92e3-48c2-b52f-e70ff30805a0\") " pod="openshift-ingress-canary/ingress-canary-fmc2z"
Sep 04 23:37:01.920606 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.920581    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9r6\" (UniqueName: \"kubernetes.io/projected/8c3a8a40-d2d2-4093-85dc-25b176df91d2-kube-api-access-cq9r6\") pod \"console-operator-85b454874b-5c7r9\" (UID: \"8c3a8a40-d2d2-4093-85dc-25b176df91d2\") " pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:01.920740 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.920722    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs5fg\" (UniqueName: \"kubernetes.io/projected/3d903250-92e3-48c2-b52f-e70ff30805a0-kube-api-access-bs5fg\") pod \"ingress-canary-fmc2z\" (UID: \"3d903250-92e3-48c2-b52f-e70ff30805a0\") " pod="openshift-ingress-canary/ingress-canary-fmc2z"
Sep 04 23:37:01.920785 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.920771    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmdmh\" (UniqueName: \"kubernetes.io/projected/9ad47c0d-7f00-4653-85fb-333ca37395d0-kube-api-access-pmdmh\") pod \"volume-data-source-validator-695c895bd6-vr466\" (UID: \"9ad47c0d-7f00-4653-85fb-333ca37395d0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-vr466"
Sep 04 23:37:01.921014 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.921000    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p89nm\" (UniqueName: \"kubernetes.io/projected/81b45393-bdad-471b-b018-b0b51925dcfe-kube-api-access-p89nm\") pod \"network-check-source-685456655f-rllx5\" (UID: \"81b45393-bdad-471b-b018-b0b51925dcfe\") " pod="openshift-network-diagnostics/network-check-source-685456655f-rllx5"
Sep 04 23:37:01.921051 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.921028    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvxdj\" (UniqueName: \"kubernetes.io/projected/51e702f1-195b-4e70-b454-ffe4228685ae-kube-api-access-wvxdj\") pod \"cluster-samples-operator-7cdcfcf76c-wfn79\" (UID: \"51e702f1-195b-4e70-b454-ffe4228685ae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"
Sep 04 23:37:01.943722 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.943571    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp"
Sep 04 23:37:01.969985 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.969957    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:01.984528 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:01.984499    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-vr466"
Sep 04 23:37:02.003211 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.003100    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-rllx5"
Sep 04 23:37:02.015281 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.013632    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmxf\" (UniqueName: \"kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf\") pod \"network-check-target-mgg4d\" (UID: \"20761599-3f21-4ba3-9a89-aceb760088a0\") " pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:37:02.015281 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.013731    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:37:02.015281 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.013787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f745f9b6-37b0-4710-a39f-282b4d388677-config-volume\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:02.015281 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.013815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f745f9b6-37b0-4710-a39f-282b4d388677-tmp-dir\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:02.015281 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.013859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:02.015281 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.013890    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmbhz\" (UniqueName: \"kubernetes.io/projected/f745f9b6-37b0-4710-a39f-282b4d388677-kube-api-access-kmbhz\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:02.015281 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.014542    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f745f9b6-37b0-4710-a39f-282b4d388677-config-volume\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:02.015281 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:02.014671    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:02.015281 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:02.014732    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret podName:e7234a20-6651-4386-8521-484e928babba nodeName:}" failed. No retries permitted until 2026-09-04 23:37:34.014713944 +0000 UTC m=+66.261381022 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret") pod "global-pull-secret-syncer-hq8l5" (UID: "e7234a20-6651-4386-8521-484e928babba") : object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:02.015281 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:02.015176    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 23:37:02.015281 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.015233    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f745f9b6-37b0-4710-a39f-282b4d388677-tmp-dir\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:02.015281 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:02.015239    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls podName:f745f9b6-37b0-4710-a39f-282b4d388677 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:02.515223929 +0000 UTC m=+34.761891001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls") pod "dns-default-rv66q" (UID: "f745f9b6-37b0-4710-a39f-282b4d388677") : secret "dns-default-metrics-tls" not found
Sep 04 23:37:02.017234 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.017215    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqmxf\" (UniqueName: \"kubernetes.io/projected/20761599-3f21-4ba3-9a89-aceb760088a0-kube-api-access-kqmxf\") pod \"network-check-target-mgg4d\" (UID: \"20761599-3f21-4ba3-9a89-aceb760088a0\") " pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:37:02.023627 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.023589    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmbhz\" (UniqueName: \"kubernetes.io/projected/f745f9b6-37b0-4710-a39f-282b4d388677-kube-api-access-kmbhz\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:02.042487 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.042102    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fmc2z"
Sep 04 23:37:02.213108 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.213078    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687"]
Sep 04 23:37:02.215950 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:02.215918    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6f2cdb_29a1_4245_83c3_1bf5eb182073.slice/crio-3d751d5b29f1cd7604631c2734ff579da9a5a651ec994941765060ab91779bca WatchSource:0}: Error finding container 3d751d5b29f1cd7604631c2734ff579da9a5a651ec994941765060ab91779bca: Status 404 returned error can't find the container with id 3d751d5b29f1cd7604631c2734ff579da9a5a651ec994941765060ab91779bca
Sep 04 23:37:02.229646 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.229620    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp"]
Sep 04 23:37:02.233010 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:02.232988    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cf8c41_30cb_4ce3_a9b1_193bdce73a60.slice/crio-c0fc4da6579ab19cd052fb938e9f5a36bac1d0a97481ba9dd290f6a5b54abf91 WatchSource:0}: Error finding container c0fc4da6579ab19cd052fb938e9f5a36bac1d0a97481ba9dd290f6a5b54abf91: Status 404 returned error can't find the container with id c0fc4da6579ab19cd052fb938e9f5a36bac1d0a97481ba9dd290f6a5b54abf91
Sep 04 23:37:02.255479 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.255453    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-5c7r9"]
Sep 04 23:37:02.316847 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.316812    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:02.317025 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:02.316950    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 04 23:37:02.317025 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:02.316966    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cdf99854b-t9dhs: secret "image-registry-tls" not found
Sep 04 23:37:02.317130 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:02.317029    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls podName:ceeabe27-d2d7-4819-aa74-cc9609f8e3e6 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:03.317009503 +0000 UTC m=+35.563676573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls") pod "image-registry-cdf99854b-t9dhs" (UID: "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6") : secret "image-registry-tls" not found
Sep 04 23:37:02.336475 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.336392    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-vr466"]
Sep 04 23:37:02.346730 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.346706    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-rllx5"]
Sep 04 23:37:02.349471 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.349448    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fmc2z"]
Sep 04 23:37:02.351773 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.351638    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:37:02.351773 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.351638    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:37:02.351887 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.351640    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:37:02.354420 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.354226    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 04 23:37:02.354420 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.354316    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 04 23:37:02.354420 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.354332    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vhhzf\""
Sep 04 23:37:02.354631 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.354443    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8jjtb\""
Sep 04 23:37:02.375666 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:02.375630    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d903250_92e3_48c2_b52f_e70ff30805a0.slice/crio-2f985f035082afe0cab1168e616e5d0abe3c3933de3d0887446b1b78caf3c3a8 WatchSource:0}: Error finding container 2f985f035082afe0cab1168e616e5d0abe3c3933de3d0887446b1b78caf3c3a8: Status 404 returned error can't find the container with id 2f985f035082afe0cab1168e616e5d0abe3c3933de3d0887446b1b78caf3c3a8
Sep 04 23:37:02.383850 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:02.383806    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3a8a40_d2d2_4093_85dc_25b176df91d2.slice/crio-e4a9ea673948f9d48abb50903443b94139bc0ea942577a4cac00b5b25bc72b41 WatchSource:0}: Error finding container e4a9ea673948f9d48abb50903443b94139bc0ea942577a4cac00b5b25bc72b41: Status 404 returned error can't find the container with id e4a9ea673948f9d48abb50903443b94139bc0ea942577a4cac00b5b25bc72b41
Sep 04 23:37:02.384114 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:02.384092    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b45393_bdad_471b_b018_b0b51925dcfe.slice/crio-a9c826ec8c279ae629ba0deaf5424f72b1f373a10907669cae24ef1c94a3bed1 WatchSource:0}: Error finding container a9c826ec8c279ae629ba0deaf5424f72b1f373a10907669cae24ef1c94a3bed1: Status 404 returned error can't find the container with id a9c826ec8c279ae629ba0deaf5424f72b1f373a10907669cae24ef1c94a3bed1
Sep 04 23:37:02.384320 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:02.384295    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad47c0d_7f00_4653_85fb_333ca37395d0.slice/crio-0323714add516a3db2ad6aebe603eea1d8bc12fed463e2e21bcc3754f32a63b5 WatchSource:0}: Error finding container 0323714add516a3db2ad6aebe603eea1d8bc12fed463e2e21bcc3754f32a63b5: Status 404 returned error can't find the container with id 0323714add516a3db2ad6aebe603eea1d8bc12fed463e2e21bcc3754f32a63b5
Sep 04 23:37:02.392476 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.392458    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:37:02.417215 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.417184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wfn79\" (UID: \"51e702f1-195b-4e70-b454-ffe4228685ae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"
Sep 04 23:37:02.417351 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:02.417333    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 04 23:37:02.417428 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:02.417409    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls podName:51e702f1-195b-4e70-b454-ffe4228685ae nodeName:}" failed. No retries permitted until 2026-09-04 23:37:03.417387577 +0000 UTC m=+35.664054651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-wfn79" (UID: "51e702f1-195b-4e70-b454-ffe4228685ae") : secret "samples-operator-tls" not found
Sep 04 23:37:02.517646 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.517616    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:02.517813 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:02.517767    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 23:37:02.517892 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:02.517837    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls podName:f745f9b6-37b0-4710-a39f-282b4d388677 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:03.517815647 +0000 UTC m=+35.764482723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls") pod "dns-default-rv66q" (UID: "f745f9b6-37b0-4710-a39f-282b4d388677") : secret "dns-default-metrics-tls" not found
Sep 04 23:37:02.518771 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.518747    2562 generic.go:356] "Generic (PLEG): container finished" podID="133ae0cb-71c7-469e-bb5e-be538e8de7b3" containerID="46c7c97d04854b3ff977a675961641aefc7d913170f3088ce3ee37e82c989754" exitCode=0
Sep 04 23:37:02.518869 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.518818    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47qkp" event={"ID":"133ae0cb-71c7-469e-bb5e-be538e8de7b3","Type":"ContainerDied","Data":"46c7c97d04854b3ff977a675961641aefc7d913170f3088ce3ee37e82c989754"}
Sep 04 23:37:02.519962 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.519939    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-vr466" event={"ID":"9ad47c0d-7f00-4653-85fb-333ca37395d0","Type":"ContainerStarted","Data":"0323714add516a3db2ad6aebe603eea1d8bc12fed463e2e21bcc3754f32a63b5"}
Sep 04 23:37:02.521045 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.521026    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-rllx5" event={"ID":"81b45393-bdad-471b-b018-b0b51925dcfe","Type":"ContainerStarted","Data":"a9c826ec8c279ae629ba0deaf5424f72b1f373a10907669cae24ef1c94a3bed1"}
Sep 04 23:37:02.522173 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.522153    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp" event={"ID":"91cf8c41-30cb-4ce3-a9b1-193bdce73a60","Type":"ContainerStarted","Data":"c0fc4da6579ab19cd052fb938e9f5a36bac1d0a97481ba9dd290f6a5b54abf91"}
Sep 04 23:37:02.523232 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.523213    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-5c7r9" event={"ID":"8c3a8a40-d2d2-4093-85dc-25b176df91d2","Type":"ContainerStarted","Data":"e4a9ea673948f9d48abb50903443b94139bc0ea942577a4cac00b5b25bc72b41"}
Sep 04 23:37:02.524156 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.524139    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fmc2z" event={"ID":"3d903250-92e3-48c2-b52f-e70ff30805a0","Type":"ContainerStarted","Data":"2f985f035082afe0cab1168e616e5d0abe3c3933de3d0887446b1b78caf3c3a8"}
Sep 04 23:37:02.525062 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.525045    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687" event={"ID":"2b6f2cdb-29a1-4245-83c3-1bf5eb182073","Type":"ContainerStarted","Data":"3d751d5b29f1cd7604631c2734ff579da9a5a651ec994941765060ab91779bca"}
Sep 04 23:37:02.669497 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:02.669260    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mgg4d"]
Sep 04 23:37:02.672642 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:02.672620    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20761599_3f21_4ba3_9a89_aceb760088a0.slice/crio-2ef4c5df3e8c4cb020372ceebfaf138933a57b1c1235656e310563706f6b5496 WatchSource:0}: Error finding container 2ef4c5df3e8c4cb020372ceebfaf138933a57b1c1235656e310563706f6b5496: Status 404 returned error can't find the container with id 2ef4c5df3e8c4cb020372ceebfaf138933a57b1c1235656e310563706f6b5496
Sep 04 23:37:03.325063 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:03.325028    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:03.325236 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:03.325188    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 04 23:37:03.325236 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:03.325204    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cdf99854b-t9dhs: secret "image-registry-tls" not found
Sep 04 23:37:03.325353 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:03.325272    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls podName:ceeabe27-d2d7-4819-aa74-cc9609f8e3e6 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:05.325254681 +0000 UTC m=+37.571921743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls") pod "image-registry-cdf99854b-t9dhs" (UID: "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6") : secret "image-registry-tls" not found
Sep 04 23:37:03.427046 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:03.426394    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wfn79\" (UID: \"51e702f1-195b-4e70-b454-ffe4228685ae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"
Sep 04 23:37:03.427046 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:03.426552    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 04 23:37:03.427046 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:03.426618    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls podName:51e702f1-195b-4e70-b454-ffe4228685ae nodeName:}" failed. No retries permitted until 2026-09-04 23:37:05.426599521 +0000 UTC m=+37.673266606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-wfn79" (UID: "51e702f1-195b-4e70-b454-ffe4228685ae") : secret "samples-operator-tls" not found
Sep 04 23:37:03.527511 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:03.526963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:03.527511 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:03.527105    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 23:37:03.527511 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:03.527167    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls podName:f745f9b6-37b0-4710-a39f-282b4d388677 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:05.527149633 +0000 UTC m=+37.773816712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls") pod "dns-default-rv66q" (UID: "f745f9b6-37b0-4710-a39f-282b4d388677") : secret "dns-default-metrics-tls" not found
Sep 04 23:37:03.539204 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:03.539173    2562 generic.go:356] "Generic (PLEG): container finished" podID="133ae0cb-71c7-469e-bb5e-be538e8de7b3" containerID="c80a54f77c594fdc37cadda6d0e23b5d49cc5f725c426bcec3d6843d4b939a23" exitCode=0
Sep 04 23:37:03.539336 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:03.539274    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47qkp" event={"ID":"133ae0cb-71c7-469e-bb5e-be538e8de7b3","Type":"ContainerDied","Data":"c80a54f77c594fdc37cadda6d0e23b5d49cc5f725c426bcec3d6843d4b939a23"}
Sep 04 23:37:03.541890 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:03.541540    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mgg4d" event={"ID":"20761599-3f21-4ba3-9a89-aceb760088a0","Type":"ContainerStarted","Data":"2ef4c5df3e8c4cb020372ceebfaf138933a57b1c1235656e310563706f6b5496"}
Sep 04 23:37:04.560302 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:04.560206    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47qkp" event={"ID":"133ae0cb-71c7-469e-bb5e-be538e8de7b3","Type":"ContainerStarted","Data":"5567dccef565ff9ce78ba2480a560b73762ee26ddc9f14b41946736fd315d667"}
Sep 04 23:37:04.582265 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:04.582210    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-47qkp" podStartSLOduration=7.256161179 podStartE2EDuration="36.582189341s" totalImagesPullingTime="29.326028162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:04.580281798 +0000 UTC m=+36.826948878" watchObservedRunningTime="2026-09-04 23:37:04.582189341 +0000 UTC m=+36.828856421"
Sep 04 23:37:04.722334 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:04.722294    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:04.807978 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:04.807883    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:05.345372 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:05.345236    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:05.345600 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:05.345392    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 04 23:37:05.345600 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:05.345413    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cdf99854b-t9dhs: secret "image-registry-tls" not found
Sep 04 23:37:05.345600 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:05.345478    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls podName:ceeabe27-d2d7-4819-aa74-cc9609f8e3e6 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:09.345456777 +0000 UTC m=+41.592123864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls") pod "image-registry-cdf99854b-t9dhs" (UID: "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6") : secret "image-registry-tls" not found
Sep 04 23:37:05.446753 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:05.446711    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wfn79\" (UID: \"51e702f1-195b-4e70-b454-ffe4228685ae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"
Sep 04 23:37:05.446917 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:05.446876    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 04 23:37:05.446974 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:05.446952    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls podName:51e702f1-195b-4e70-b454-ffe4228685ae nodeName:}" failed. No retries permitted until 2026-09-04 23:37:09.446932026 +0000 UTC m=+41.693599088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-wfn79" (UID: "51e702f1-195b-4e70-b454-ffe4228685ae") : secret "samples-operator-tls" not found
Sep 04 23:37:05.547700 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:05.547646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:05.547865 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:05.547815    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 23:37:05.547918 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:05.547894    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls podName:f745f9b6-37b0-4710-a39f-282b4d388677 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:09.547876301 +0000 UTC m=+41.794543381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls") pod "dns-default-rv66q" (UID: "f745f9b6-37b0-4710-a39f-282b4d388677") : secret "dns-default-metrics-tls" not found
Sep 04 23:37:05.566933 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:05.566887    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-vr466" event={"ID":"9ad47c0d-7f00-4653-85fb-333ca37395d0","Type":"ContainerStarted","Data":"ec175cebdffcd19525911dde974913004b88764e6f27f16c8702994f4486f35a"}
Sep 04 23:37:05.581815 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:05.581325    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-vr466" podStartSLOduration=27.461850671 podStartE2EDuration="29.581311296s" totalImagesPullingTime="2.119460625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:05.581233111 +0000 UTC m=+37.827900190" watchObservedRunningTime="2026-09-04 23:37:05.581311296 +0000 UTC m=+37.827978368"
Sep 04 23:37:05.655356 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:05.655258    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:05.708242 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:05.708153    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:06.624760 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:06.624727    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:06.676323 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:06.676289    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:09.377873 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:09.377792    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:09.378236 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:09.377913    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 04 23:37:09.378236 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:09.377924    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cdf99854b-t9dhs: secret "image-registry-tls" not found
Sep 04 23:37:09.378236 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:09.377972    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls podName:ceeabe27-d2d7-4819-aa74-cc9609f8e3e6 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:17.377960026 +0000 UTC m=+49.624627084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls") pod "image-registry-cdf99854b-t9dhs" (UID: "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6") : secret "image-registry-tls" not found
Sep 04 23:37:09.478476 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:09.478446    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wfn79\" (UID: \"51e702f1-195b-4e70-b454-ffe4228685ae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"
Sep 04 23:37:09.478612 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:09.478593    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 04 23:37:09.478693 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:09.478681    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls podName:51e702f1-195b-4e70-b454-ffe4228685ae nodeName:}" failed. No retries permitted until 2026-09-04 23:37:17.478641857 +0000 UTC m=+49.725308914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-wfn79" (UID: "51e702f1-195b-4e70-b454-ffe4228685ae") : secret "samples-operator-tls" not found
Sep 04 23:37:09.579363 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:09.579337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:09.582859 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:09.579840    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 23:37:09.582859 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:09.579919    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls podName:f745f9b6-37b0-4710-a39f-282b4d388677 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:17.579900863 +0000 UTC m=+49.826567936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls") pod "dns-default-rv66q" (UID: "f745f9b6-37b0-4710-a39f-282b4d388677") : secret "dns-default-metrics-tls" not found
Sep 04 23:37:10.489193 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.489156    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:10.537680 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.537573    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:10.584253 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.583981    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-rllx5" event={"ID":"81b45393-bdad-471b-b018-b0b51925dcfe","Type":"ContainerStarted","Data":"f5fdf7ab90bb12143c88761a7bcc4a59a056d025a72b27546725dbbc32aeb335"}
Sep 04 23:37:10.585679 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.585636    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp" event={"ID":"91cf8c41-30cb-4ce3-a9b1-193bdce73a60","Type":"ContainerStarted","Data":"12fcb68aade414abc5617266689146f2c78e3d796f3363c5a99befc23f37f91d"}
Sep 04 23:37:10.587691 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.587671    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/0.log"
Sep 04 23:37:10.587810 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.587707    2562 generic.go:356] "Generic (PLEG): container finished" podID="8c3a8a40-d2d2-4093-85dc-25b176df91d2" containerID="cf2a18e1912ee940c92707858ffc31bb9581121f87e83eef9052ebdda3fdcaa8" exitCode=255
Sep 04 23:37:10.587810 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.587737    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-5c7r9" event={"ID":"8c3a8a40-d2d2-4093-85dc-25b176df91d2","Type":"ContainerDied","Data":"cf2a18e1912ee940c92707858ffc31bb9581121f87e83eef9052ebdda3fdcaa8"}
Sep 04 23:37:10.587953 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.587938    2562 scope.go:117] "RemoveContainer" containerID="cf2a18e1912ee940c92707858ffc31bb9581121f87e83eef9052ebdda3fdcaa8"
Sep 04 23:37:10.589462 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.589438    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mgg4d" event={"ID":"20761599-3f21-4ba3-9a89-aceb760088a0","Type":"ContainerStarted","Data":"e2d6dd51afbbf453ed03117022c938dc3c524e6651d3d986b8b11d9db66d9ed0"}
Sep 04 23:37:10.589569 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.589519    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:37:10.591226 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.591205    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fmc2z" event={"ID":"3d903250-92e3-48c2-b52f-e70ff30805a0","Type":"ContainerStarted","Data":"98b3a045d18a95779728d62cf479cb95c7e91659cc4066a653897c5fc4162378"}
Sep 04 23:37:10.592836 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.592815    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687" event={"ID":"2b6f2cdb-29a1-4245-83c3-1bf5eb182073","Type":"ContainerStarted","Data":"ee7eb28243b322574a4d18a0294036de0dcb97e4c11e8830acf1a32f746c60ce"}
Sep 04 23:37:10.600939 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.600895    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-rllx5" podStartSLOduration=15.469124283 podStartE2EDuration="22.600878348s" totalImagesPullingTime="7.131754065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:10.600008671 +0000 UTC m=+42.846675751" watchObservedRunningTime="2026-09-04 23:37:10.600878348 +0000 UTC m=+42.847545428"
Sep 04 23:37:10.615708 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.615642    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687" podStartSLOduration=27.350043641 podStartE2EDuration="34.615625653s" totalImagesPullingTime="7.265582012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:10.615377409 +0000 UTC m=+42.862044489" watchObservedRunningTime="2026-09-04 23:37:10.615625653 +0000 UTC m=+42.862292733"
Sep 04 23:37:10.629253 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.628960    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp" podStartSLOduration=27.355651591 podStartE2EDuration="34.628942027s" totalImagesPullingTime="7.273290436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:10.628683383 +0000 UTC m=+42.875350464" watchObservedRunningTime="2026-09-04 23:37:10.628942027 +0000 UTC m=+42.875609107"
Sep 04 23:37:10.662024 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.661974    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fmc2z" podStartSLOduration=2.531157011 podStartE2EDuration="9.661959059s" totalImagesPullingTime="7.130802048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:10.660643757 +0000 UTC m=+42.907310835" watchObservedRunningTime="2026-09-04 23:37:10.661959059 +0000 UTC m=+42.908626138"
Sep 04 23:37:10.674319 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:10.674277    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mgg4d" podStartSLOduration=35.84093347 podStartE2EDuration="42.674261574s" totalImagesPullingTime="6.833328104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:10.674100428 +0000 UTC m=+42.920767508" watchObservedRunningTime="2026-09-04 23:37:10.674261574 +0000 UTC m=+42.920928652"
Sep 04 23:37:11.010378 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.010292    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:11.458518 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.458484    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:11.562218 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.562185    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:11.597780 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.597755    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 04 23:37:11.598215 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.598197    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/0.log"
Sep 04 23:37:11.598323 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.598236    2562 generic.go:356] "Generic (PLEG): container finished" podID="8c3a8a40-d2d2-4093-85dc-25b176df91d2" containerID="c8f40a395d20836af12a8e25dacdee0eeec66bab686688fd02b054519593cbf2" exitCode=255
Sep 04 23:37:11.598387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.598335    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-5c7r9" event={"ID":"8c3a8a40-d2d2-4093-85dc-25b176df91d2","Type":"ContainerDied","Data":"c8f40a395d20836af12a8e25dacdee0eeec66bab686688fd02b054519593cbf2"}
Sep 04 23:37:11.598442 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.598385    2562 scope.go:117] "RemoveContainer" containerID="cf2a18e1912ee940c92707858ffc31bb9581121f87e83eef9052ebdda3fdcaa8"
Sep 04 23:37:11.598521 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.598500    2562 scope.go:117] "RemoveContainer" containerID="c8f40a395d20836af12a8e25dacdee0eeec66bab686688fd02b054519593cbf2"
Sep 04 23:37:11.598793 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:11.598734    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-5c7r9_openshift-console-operator(8c3a8a40-d2d2-4093-85dc-25b176df91d2)\"" pod="openshift-console-operator/console-operator-85b454874b-5c7r9" podUID="8c3a8a40-d2d2-4093-85dc-25b176df91d2"
Sep 04 23:37:11.721773 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.721677    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:11.822092 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.822055    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:11.856687 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.856636    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-n22lt"]
Sep 04 23:37:11.881945 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.881914    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-n22lt"]
Sep 04 23:37:11.882125 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.882014    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-n22lt"
Sep 04 23:37:11.884829 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.884808    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 04 23:37:11.885094 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.885075    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 04 23:37:11.885925 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.885904    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jxvkg\""
Sep 04 23:37:11.901606 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.901572    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:11.954626 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.954594    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:11.970602 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.970570    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:11.970743 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.970631    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:11.998006 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:11.997932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqnf9\" (UniqueName: \"kubernetes.io/projected/506ee782-a460-4b93-afae-6a09bc31e2d3-kube-api-access-wqnf9\") pod \"migrator-fdbb8fdff-n22lt\" (UID: \"506ee782-a460-4b93-afae-6a09bc31e2d3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-n22lt"
Sep 04 23:37:12.098564 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:12.098525    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqnf9\" (UniqueName: \"kubernetes.io/projected/506ee782-a460-4b93-afae-6a09bc31e2d3-kube-api-access-wqnf9\") pod \"migrator-fdbb8fdff-n22lt\" (UID: \"506ee782-a460-4b93-afae-6a09bc31e2d3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-n22lt"
Sep 04 23:37:12.119845 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:12.119825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqnf9\" (UniqueName: \"kubernetes.io/projected/506ee782-a460-4b93-afae-6a09bc31e2d3-kube-api-access-wqnf9\") pod \"migrator-fdbb8fdff-n22lt\" (UID: \"506ee782-a460-4b93-afae-6a09bc31e2d3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-n22lt"
Sep 04 23:37:12.191549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:12.191518    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-n22lt"
Sep 04 23:37:12.479011 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:12.478983    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-n22lt"]
Sep 04 23:37:12.481692 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:12.481642    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506ee782_a460_4b93_afae_6a09bc31e2d3.slice/crio-eb0fa8c88def3081796787f04ca25788a4fc192f9821a1887811c0240c4b0c25 WatchSource:0}: Error finding container eb0fa8c88def3081796787f04ca25788a4fc192f9821a1887811c0240c4b0c25: Status 404 returned error can't find the container with id eb0fa8c88def3081796787f04ca25788a4fc192f9821a1887811c0240c4b0c25
Sep 04 23:37:12.602802 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:12.602728    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 04 23:37:12.603186 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:12.603049    2562 scope.go:117] "RemoveContainer" containerID="c8f40a395d20836af12a8e25dacdee0eeec66bab686688fd02b054519593cbf2"
Sep 04 23:37:12.603238 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:12.603214    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-5c7r9_openshift-console-operator(8c3a8a40-d2d2-4093-85dc-25b176df91d2)\"" pod="openshift-console-operator/console-operator-85b454874b-5c7r9" podUID="8c3a8a40-d2d2-4093-85dc-25b176df91d2"
Sep 04 23:37:12.604023 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:12.603999    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-n22lt" event={"ID":"506ee782-a460-4b93-afae-6a09bc31e2d3","Type":"ContainerStarted","Data":"eb0fa8c88def3081796787f04ca25788a4fc192f9821a1887811c0240c4b0c25"}
Sep 04 23:37:13.315434 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.315404    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-dlgwg"]
Sep 04 23:37:13.327363 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.327276    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-dlgwg"
Sep 04 23:37:13.328358 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.328320    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-dlgwg"]
Sep 04 23:37:13.329630 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.329470    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 04 23:37:13.329630 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.329505    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 04 23:37:13.329630 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.329512    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 04 23:37:13.329836 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.329724    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 04 23:37:13.330533 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.330411    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-p9t6t\""
Sep 04 23:37:13.349178 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.349148    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:13.400537 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.400503    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:13.408413 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.408385    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f-signing-cabundle\") pod \"service-ca-75d99c484-dlgwg\" (UID: \"b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f\") " pod="openshift-service-ca/service-ca-75d99c484-dlgwg"
Sep 04 23:37:13.408555 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.408451    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm9mw\" (UniqueName: \"kubernetes.io/projected/b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f-kube-api-access-vm9mw\") pod \"service-ca-75d99c484-dlgwg\" (UID: \"b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f\") " pod="openshift-service-ca/service-ca-75d99c484-dlgwg"
Sep 04 23:37:13.408615 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.408590    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f-signing-key\") pod \"service-ca-75d99c484-dlgwg\" (UID: \"b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f\") " pod="openshift-service-ca/service-ca-75d99c484-dlgwg"
Sep 04 23:37:13.509306 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.509266    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f-signing-cabundle\") pod \"service-ca-75d99c484-dlgwg\" (UID: \"b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f\") " pod="openshift-service-ca/service-ca-75d99c484-dlgwg"
Sep 04 23:37:13.509496 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.509444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm9mw\" (UniqueName: \"kubernetes.io/projected/b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f-kube-api-access-vm9mw\") pod \"service-ca-75d99c484-dlgwg\" (UID: \"b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f\") " pod="openshift-service-ca/service-ca-75d99c484-dlgwg"
Sep 04 23:37:13.509593 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.509575    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f-signing-key\") pod \"service-ca-75d99c484-dlgwg\" (UID: \"b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f\") " pod="openshift-service-ca/service-ca-75d99c484-dlgwg"
Sep 04 23:37:13.510192 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.510163    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f-signing-cabundle\") pod \"service-ca-75d99c484-dlgwg\" (UID: \"b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f\") " pod="openshift-service-ca/service-ca-75d99c484-dlgwg"
Sep 04 23:37:13.512376 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.512354    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f-signing-key\") pod \"service-ca-75d99c484-dlgwg\" (UID: \"b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f\") " pod="openshift-service-ca/service-ca-75d99c484-dlgwg"
Sep 04 23:37:13.517497 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.517434    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm9mw\" (UniqueName: \"kubernetes.io/projected/b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f-kube-api-access-vm9mw\") pod \"service-ca-75d99c484-dlgwg\" (UID: \"b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f\") " pod="openshift-service-ca/service-ca-75d99c484-dlgwg"
Sep 04 23:37:13.608325 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.608264    2562 scope.go:117] "RemoveContainer" containerID="c8f40a395d20836af12a8e25dacdee0eeec66bab686688fd02b054519593cbf2"
Sep 04 23:37:13.608671 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:13.608418    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-5c7r9_openshift-console-operator(8c3a8a40-d2d2-4093-85dc-25b176df91d2)\"" pod="openshift-console-operator/console-operator-85b454874b-5c7r9" podUID="8c3a8a40-d2d2-4093-85dc-25b176df91d2"
Sep 04 23:37:13.638812 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.638792    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-dlgwg"
Sep 04 23:37:13.916460 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:13.916394    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-dlgwg"]
Sep 04 23:37:13.934605 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:13.934578    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d51c66_9e6b_4c9d_a2ab_71e33b37c14f.slice/crio-f816d0d8f215ee5a0bb71633349b7e0fce4dea3adb0b5e4e9c0a7fd39cf27732 WatchSource:0}: Error finding container f816d0d8f215ee5a0bb71633349b7e0fce4dea3adb0b5e4e9c0a7fd39cf27732: Status 404 returned error can't find the container with id f816d0d8f215ee5a0bb71633349b7e0fce4dea3adb0b5e4e9c0a7fd39cf27732
Sep 04 23:37:14.188356 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:14.188313    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log" limit=1024
Sep 04 23:37:14.612809 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:14.612763    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-n22lt" event={"ID":"506ee782-a460-4b93-afae-6a09bc31e2d3","Type":"ContainerStarted","Data":"e0f3b6b8a7f651d6a8375ba818e9d7921e7556ff66184f14990add4ff1fd6093"}
Sep 04 23:37:14.613305 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:14.612871    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-n22lt" event={"ID":"506ee782-a460-4b93-afae-6a09bc31e2d3","Type":"ContainerStarted","Data":"48c70a16b9130f9f27ecb7a9a54874b980a10acc369e804c13527c44a899da7e"}
Sep 04 23:37:14.614377 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:14.614352    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-dlgwg" event={"ID":"b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f","Type":"ContainerStarted","Data":"692165c3b37f89fab240445ed762ca0136b552799252c49bb645c0f7bebafd39"}
Sep 04 23:37:14.614458 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:14.614395    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-dlgwg" event={"ID":"b5d51c66-9e6b-4c9d-a2ab-71e33b37c14f","Type":"ContainerStarted","Data":"f816d0d8f215ee5a0bb71633349b7e0fce4dea3adb0b5e4e9c0a7fd39cf27732"}
Sep 04 23:37:14.629228 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:14.629186    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-n22lt" podStartSLOduration=2.154659109 podStartE2EDuration="3.62917302s" totalImagesPullingTime="1.474513911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:37:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:14.627685017 +0000 UTC m=+46.874352095" watchObservedRunningTime="2026-09-04 23:37:14.62917302 +0000 UTC m=+46.875840100"
Sep 04 23:37:14.641971 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:14.641931    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-dlgwg" podStartSLOduration=1.641917191 podStartE2EDuration="1.641917191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:14.6413456 +0000 UTC m=+46.888012679" watchObservedRunningTime="2026-09-04 23:37:14.641917191 +0000 UTC m=+46.888584261"
Sep 04 23:37:17.443734 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:17.443693    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:17.444111 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:17.443809    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 04 23:37:17.444111 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:17.443821    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cdf99854b-t9dhs: secret "image-registry-tls" not found
Sep 04 23:37:17.444111 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:17.443866    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls podName:ceeabe27-d2d7-4819-aa74-cc9609f8e3e6 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:33.443853608 +0000 UTC m=+65.690520664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls") pod "image-registry-cdf99854b-t9dhs" (UID: "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6") : secret "image-registry-tls" not found
Sep 04 23:37:17.544912 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:17.544867    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wfn79\" (UID: \"51e702f1-195b-4e70-b454-ffe4228685ae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"
Sep 04 23:37:17.547254 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:17.547224    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51e702f1-195b-4e70-b454-ffe4228685ae-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wfn79\" (UID: \"51e702f1-195b-4e70-b454-ffe4228685ae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"
Sep 04 23:37:17.620065 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:17.620032    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"
Sep 04 23:37:17.646228 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:17.646198    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:17.646369 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:17.646343    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 23:37:17.646429 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:17.646418    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls podName:f745f9b6-37b0-4710-a39f-282b4d388677 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:33.646401909 +0000 UTC m=+65.893068970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls") pod "dns-default-rv66q" (UID: "f745f9b6-37b0-4710-a39f-282b4d388677") : secret "dns-default-metrics-tls" not found
Sep 04 23:37:17.918223 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:17.918191    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79"]
Sep 04 23:37:18.629810 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:18.629773    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79" event={"ID":"51e702f1-195b-4e70-b454-ffe4228685ae","Type":"ContainerStarted","Data":"30ca6518183b7f5d986c15d3c4252890f1f40be734496e13566ee860ce4508e6"}
Sep 04 23:37:20.637801 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:20.637762    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79" event={"ID":"51e702f1-195b-4e70-b454-ffe4228685ae","Type":"ContainerStarted","Data":"25f2bd0f0e93b6751e80de9a3b1789070421b02f691a1a8050905309348fa11d"}
Sep 04 23:37:20.638222 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:20.637827    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79" event={"ID":"51e702f1-195b-4e70-b454-ffe4228685ae","Type":"ContainerStarted","Data":"77a13df59fe462f5b93ffa951d4192f6f0e46227f17b91f1f82c008ab11d6701"}
Sep 04 23:37:20.654069 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:20.654026    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wfn79" podStartSLOduration=43.022524009 podStartE2EDuration="44.654010936s" totalImagesPullingTime="1.631486927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:20.652804131 +0000 UTC m=+52.899471210" watchObservedRunningTime="2026-09-04 23:37:20.654010936 +0000 UTC m=+52.900678016"
Sep 04 23:37:27.515534 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:27.515505    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d75ld"
Sep 04 23:37:29.351982 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:29.351952    2562 scope.go:117] "RemoveContainer" containerID="c8f40a395d20836af12a8e25dacdee0eeec66bab686688fd02b054519593cbf2"
Sep 04 23:37:29.669269 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:29.669195    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 04 23:37:29.669394 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:29.669270    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-5c7r9" event={"ID":"8c3a8a40-d2d2-4093-85dc-25b176df91d2","Type":"ContainerStarted","Data":"0aa8f6550489a02f647abbcd948a2acbe3b821368adb1f21042b2781af52ba21"}
Sep 04 23:37:29.669562 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:29.669545    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:29.684816 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:29.684694    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-5c7r9" podStartSLOduration=46.562544397 podStartE2EDuration="53.684675983s" totalImagesPullingTime="7.122131586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:29.684258016 +0000 UTC m=+61.930925089" watchObservedRunningTime="2026-09-04 23:37:29.684675983 +0000 UTC m=+61.931343061"
Sep 04 23:37:30.183045 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:30.183016    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-5c7r9"
Sep 04 23:37:32.032129 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.032094    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-vxcx6"]
Sep 04 23:37:32.061816 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.061787    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-6fj75"]
Sep 04 23:37:32.061973 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.061916    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.064718 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.064694    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-k7kr4\""
Sep 04 23:37:32.065194 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.065167    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 04 23:37:32.065689 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.065499    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 04 23:37:32.065689 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.065620    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 04 23:37:32.066077 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.066052    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 04 23:37:32.073355 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.073337    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 04 23:37:32.075478 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.075461    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9"]
Sep 04 23:37:32.075580 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.075565    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-6fj75"
Sep 04 23:37:32.077784 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.077752    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 04 23:37:32.077880 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.077778    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xdxvl\""
Sep 04 23:37:32.077880 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.077853    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 04 23:37:32.089698 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.089677    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"]
Sep 04 23:37:32.089824 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.089804    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9"
Sep 04 23:37:32.092240 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.092223    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 04 23:37:32.093171 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.093156    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 04 23:37:32.093255 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.093193    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-t8ztt\""
Sep 04 23:37:32.093255 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.093225    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 04 23:37:32.093438 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.093424    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 04 23:37:32.111862 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.111816    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"]
Sep 04 23:37:32.112243 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.112217    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"
Sep 04 23:37:32.114976 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.114957    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 04 23:37:32.115074 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.114975    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 04 23:37:32.115074 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.114993    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 04 23:37:32.115074 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.115068    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 04 23:37:32.131109 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.131089    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-6fj75"]
Sep 04 23:37:32.131194 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.131147    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-vxcx6"]
Sep 04 23:37:32.131194 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.131168    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"]
Sep 04 23:37:32.131194 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.131189    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9"]
Sep 04 23:37:32.131313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.131206    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"]
Sep 04 23:37:32.131313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.131222    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.131407 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.131225    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c849d556-mcx9f"]
Sep 04 23:37:32.133601 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.133582    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 04 23:37:32.133883 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.133863    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 04 23:37:32.134762 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.134745    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 04 23:37:32.134875 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.134840    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 04 23:37:32.149095 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.149046    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c849d556-mcx9f"]
Sep 04 23:37:32.149308 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.149280    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c849d556-mcx9f"
Sep 04 23:37:32.152082 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.152060    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-j4tgw\""
Sep 04 23:37:32.152203 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.152185    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 04 23:37:32.162001 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.161978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-serving-cert\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.162116 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162013    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6fdc957b-b8aa-4522-b457-cf882a06bf3c-hub\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.162116 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162039    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d8f556f98-8dd7q\" (UID: \"ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"
Sep 04 23:37:32.162116 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162066    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a55b5a2c-cebb-4737-bbb3-afc50bd85e78-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65c849d556-mcx9f\" (UID: \"a55b5a2c-cebb-4737-bbb3-afc50bd85e78\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c849d556-mcx9f"
Sep 04 23:37:32.162250 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162165    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2tmx\" (UniqueName: \"kubernetes.io/projected/ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66-kube-api-access-d2tmx\") pod \"klusterlet-addon-workmgr-5d8f556f98-8dd7q\" (UID: \"ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"
Sep 04 23:37:32.162250 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162242    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fa30820-fda2-4e06-80a1-6e9de51833ea-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-6fj75\" (UID: \"3fa30820-fda2-4e06-80a1-6e9de51833ea\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6fj75"
Sep 04 23:37:32.162330 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6fdc957b-b8aa-4522-b457-cf882a06bf3c-ca\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.162330 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6cmx\" (UniqueName: \"kubernetes.io/projected/6fdc957b-b8aa-4522-b457-cf882a06bf3c-kube-api-access-t6cmx\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.162330 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.162441 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162346    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgpr6\" (UniqueName: \"kubernetes.io/projected/08ef1223-76e2-4834-af84-b9ca9a19b579-kube-api-access-rgpr6\") pod \"cluster-monitoring-operator-6cdcc96965-vd6n9\" (UID: \"08ef1223-76e2-4834-af84-b9ca9a19b579\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9"
Sep 04 23:37:32.162441 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fdc957b-b8aa-4522-b457-cf882a06bf3c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.162441 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162402    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66-tmp\") pod \"klusterlet-addon-workmgr-5d8f556f98-8dd7q\" (UID: \"ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"
Sep 04 23:37:32.162552 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.162552 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162475    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/08ef1223-76e2-4834-af84-b9ca9a19b579-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-vd6n9\" (UID: \"08ef1223-76e2-4834-af84-b9ca9a19b579\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9"
Sep 04 23:37:32.162642 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k58j8\" (UniqueName: \"kubernetes.io/projected/a55b5a2c-cebb-4737-bbb3-afc50bd85e78-kube-api-access-k58j8\") pod \"managed-serviceaccount-addon-agent-65c849d556-mcx9f\" (UID: \"a55b5a2c-cebb-4737-bbb3-afc50bd85e78\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c849d556-mcx9f"
Sep 04 23:37:32.162642 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162601    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fa30820-fda2-4e06-80a1-6e9de51833ea-nginx-conf\") pod \"networking-console-plugin-79977f9844-6fj75\" (UID: \"3fa30820-fda2-4e06-80a1-6e9de51833ea\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6fj75"
Sep 04 23:37:32.162642 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162618    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6fdc957b-b8aa-4522-b457-cf882a06bf3c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.162766 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162698    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6fdc957b-b8aa-4522-b457-cf882a06bf3c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.162766 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx6bf\" (UniqueName: \"kubernetes.io/projected/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-kube-api-access-cx6bf\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.162873 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162776    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-tmp\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.162873 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162810    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-snapshots\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.162873 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.162826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08ef1223-76e2-4834-af84-b9ca9a19b579-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-vd6n9\" (UID: \"08ef1223-76e2-4834-af84-b9ca9a19b579\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9"
Sep 04 23:37:32.166807 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.166787    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-76b88fd7dd-5knhq"]
Sep 04 23:37:32.189661 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.189638    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-td8gm"]
Sep 04 23:37:32.189755 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.189727    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.195939 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.195919    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 04 23:37:32.196196 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.196177    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 04 23:37:32.196447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.196431    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 04 23:37:32.196850 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.196833    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-blt4m\""
Sep 04 23:37:32.197137 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.197118    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 04 23:37:32.198445 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.198213    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 04 23:37:32.198445 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.198354    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 04 23:37:32.203949 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.203929    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-td8gm"]
Sep 04 23:37:32.204057 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.203954    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76b88fd7dd-5knhq"]
Sep 04 23:37:32.204057 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.204016    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-td8gm"
Sep 04 23:37:32.206381 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.206363    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 04 23:37:32.206475 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.206450    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-5whqx\""
Sep 04 23:37:32.206799 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.206781    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 04 23:37:32.263261 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6fdc957b-b8aa-4522-b457-cf882a06bf3c-ca\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.263406 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6cmx\" (UniqueName: \"kubernetes.io/projected/6fdc957b-b8aa-4522-b457-cf882a06bf3c-kube-api-access-t6cmx\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.263406 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.263406 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263326    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgpr6\" (UniqueName: \"kubernetes.io/projected/08ef1223-76e2-4834-af84-b9ca9a19b579-kube-api-access-rgpr6\") pod \"cluster-monitoring-operator-6cdcc96965-vd6n9\" (UID: \"08ef1223-76e2-4834-af84-b9ca9a19b579\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9"
Sep 04 23:37:32.263406 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263357    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1b11f31d-54cf-49b7-a4b7-ae4984221a56-stats-auth\") pod \"router-default-76b88fd7dd-5knhq\" (UID: \"1b11f31d-54cf-49b7-a4b7-ae4984221a56\") " pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.263406 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fdc957b-b8aa-4522-b457-cf882a06bf3c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.263694 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263412    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66-tmp\") pod \"klusterlet-addon-workmgr-5d8f556f98-8dd7q\" (UID: \"ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"
Sep 04 23:37:32.263694 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.263694 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263521    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/08ef1223-76e2-4834-af84-b9ca9a19b579-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-vd6n9\" (UID: \"08ef1223-76e2-4834-af84-b9ca9a19b579\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9"
Sep 04 23:37:32.263694 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1b11f31d-54cf-49b7-a4b7-ae4984221a56-default-certificate\") pod \"router-default-76b88fd7dd-5knhq\" (UID: \"1b11f31d-54cf-49b7-a4b7-ae4984221a56\") " pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.263694 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5wdn\" (UniqueName: \"kubernetes.io/projected/1b11f31d-54cf-49b7-a4b7-ae4984221a56-kube-api-access-t5wdn\") pod \"router-default-76b88fd7dd-5knhq\" (UID: \"1b11f31d-54cf-49b7-a4b7-ae4984221a56\") " pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.263956 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263701    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k58j8\" (UniqueName: \"kubernetes.io/projected/a55b5a2c-cebb-4737-bbb3-afc50bd85e78-kube-api-access-k58j8\") pod \"managed-serviceaccount-addon-agent-65c849d556-mcx9f\" (UID: \"a55b5a2c-cebb-4737-bbb3-afc50bd85e78\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c849d556-mcx9f"
Sep 04 23:37:32.263956 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263737    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fa30820-fda2-4e06-80a1-6e9de51833ea-nginx-conf\") pod \"networking-console-plugin-79977f9844-6fj75\" (UID: \"3fa30820-fda2-4e06-80a1-6e9de51833ea\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6fj75"
Sep 04 23:37:32.263956 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263761    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6fdc957b-b8aa-4522-b457-cf882a06bf3c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.263956 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263791    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6fdc957b-b8aa-4522-b457-cf882a06bf3c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.263956 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263821    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b11f31d-54cf-49b7-a4b7-ae4984221a56-service-ca-bundle\") pod \"router-default-76b88fd7dd-5knhq\" (UID: \"1b11f31d-54cf-49b7-a4b7-ae4984221a56\") " pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.263956 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx6bf\" (UniqueName: \"kubernetes.io/projected/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-kube-api-access-cx6bf\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.263956 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263898    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-tmp\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.263956 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263922    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-snapshots\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.263956 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263949    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08ef1223-76e2-4834-af84-b9ca9a19b579-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-vd6n9\" (UID: \"08ef1223-76e2-4834-af84-b9ca9a19b579\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9"
Sep 04 23:37:32.264380 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.264003    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b11f31d-54cf-49b7-a4b7-ae4984221a56-metrics-certs\") pod \"router-default-76b88fd7dd-5knhq\" (UID: \"1b11f31d-54cf-49b7-a4b7-ae4984221a56\") " pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.264380 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.264031    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-serving-cert\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.264380 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.264054    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6fdc957b-b8aa-4522-b457-cf882a06bf3c-hub\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.264380 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.264087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d8f556f98-8dd7q\" (UID: \"ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"
Sep 04 23:37:32.264380 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.264116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a55b5a2c-cebb-4737-bbb3-afc50bd85e78-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65c849d556-mcx9f\" (UID: \"a55b5a2c-cebb-4737-bbb3-afc50bd85e78\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c849d556-mcx9f"
Sep 04 23:37:32.264380 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.264150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2tmx\" (UniqueName: \"kubernetes.io/projected/ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66-kube-api-access-d2tmx\") pod \"klusterlet-addon-workmgr-5d8f556f98-8dd7q\" (UID: \"ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"
Sep 04 23:37:32.264380 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.264201    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92grx\" (UniqueName: \"kubernetes.io/projected/082f581e-6378-4981-bba8-fd66fa8923e1-kube-api-access-92grx\") pod \"downloads-85d98c998f-td8gm\" (UID: \"082f581e-6378-4981-bba8-fd66fa8923e1\") " pod="openshift-console/downloads-85d98c998f-td8gm"
Sep 04 23:37:32.264380 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.264234    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fa30820-fda2-4e06-80a1-6e9de51833ea-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-6fj75\" (UID: \"3fa30820-fda2-4e06-80a1-6e9de51833ea\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6fj75"
Sep 04 23:37:32.264775 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.264554    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-tmp\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.264775 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.264670    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fa30820-fda2-4e06-80a1-6e9de51833ea-nginx-conf\") pod \"networking-console-plugin-79977f9844-6fj75\" (UID: \"3fa30820-fda2-4e06-80a1-6e9de51833ea\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6fj75"
Sep 04 23:37:32.264872 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.264789    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.264922 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.264889    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fdc957b-b8aa-4522-b457-cf882a06bf3c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.264922 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.263819    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66-tmp\") pod \"klusterlet-addon-workmgr-5d8f556f98-8dd7q\" (UID: \"ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"
Sep 04 23:37:32.265169 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.265124    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-snapshots\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.265676 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.265616    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/08ef1223-76e2-4834-af84-b9ca9a19b579-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-vd6n9\" (UID: \"08ef1223-76e2-4834-af84-b9ca9a19b579\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9"
Sep 04 23:37:32.266019 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.265996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.266567 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.266491    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6fdc957b-b8aa-4522-b457-cf882a06bf3c-ca\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.267359 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.267314    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fa30820-fda2-4e06-80a1-6e9de51833ea-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-6fj75\" (UID: \"3fa30820-fda2-4e06-80a1-6e9de51833ea\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6fj75"
Sep 04 23:37:32.267683 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.267646    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d8f556f98-8dd7q\" (UID: \"ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"
Sep 04 23:37:32.268220 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.268192    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6fdc957b-b8aa-4522-b457-cf882a06bf3c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.268846 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.268823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6fdc957b-b8aa-4522-b457-cf882a06bf3c-hub\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.268925 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.268895    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-serving-cert\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.269307 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.269269    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08ef1223-76e2-4834-af84-b9ca9a19b579-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-vd6n9\" (UID: \"08ef1223-76e2-4834-af84-b9ca9a19b579\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9"
Sep 04 23:37:32.269469 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.269452    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a55b5a2c-cebb-4737-bbb3-afc50bd85e78-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65c849d556-mcx9f\" (UID: \"a55b5a2c-cebb-4737-bbb3-afc50bd85e78\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c849d556-mcx9f"
Sep 04 23:37:32.269851 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.269832    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6fdc957b-b8aa-4522-b457-cf882a06bf3c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.280389 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.280357    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2tmx\" (UniqueName: \"kubernetes.io/projected/ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66-kube-api-access-d2tmx\") pod \"klusterlet-addon-workmgr-5d8f556f98-8dd7q\" (UID: \"ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"
Sep 04 23:37:32.281432 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.281340    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx6bf\" (UniqueName: \"kubernetes.io/projected/b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00-kube-api-access-cx6bf\") pod \"insights-operator-76bcd6ccf5-vxcx6\" (UID: \"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.282802 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.282753    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6cmx\" (UniqueName: \"kubernetes.io/projected/6fdc957b-b8aa-4522-b457-cf882a06bf3c-kube-api-access-t6cmx\") pod \"cluster-proxy-proxy-agent-7c969d7d87-5k6qn\" (UID: \"6fdc957b-b8aa-4522-b457-cf882a06bf3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.282899 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.282878    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgpr6\" (UniqueName: \"kubernetes.io/projected/08ef1223-76e2-4834-af84-b9ca9a19b579-kube-api-access-rgpr6\") pod \"cluster-monitoring-operator-6cdcc96965-vd6n9\" (UID: \"08ef1223-76e2-4834-af84-b9ca9a19b579\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9"
Sep 04 23:37:32.283637 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.283616    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k58j8\" (UniqueName: \"kubernetes.io/projected/a55b5a2c-cebb-4737-bbb3-afc50bd85e78-kube-api-access-k58j8\") pod \"managed-serviceaccount-addon-agent-65c849d556-mcx9f\" (UID: \"a55b5a2c-cebb-4737-bbb3-afc50bd85e78\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c849d556-mcx9f"
Sep 04 23:37:32.364976 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.364945    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92grx\" (UniqueName: \"kubernetes.io/projected/082f581e-6378-4981-bba8-fd66fa8923e1-kube-api-access-92grx\") pod \"downloads-85d98c998f-td8gm\" (UID: \"082f581e-6378-4981-bba8-fd66fa8923e1\") " pod="openshift-console/downloads-85d98c998f-td8gm"
Sep 04 23:37:32.365178 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.364992    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1b11f31d-54cf-49b7-a4b7-ae4984221a56-stats-auth\") pod \"router-default-76b88fd7dd-5knhq\" (UID: \"1b11f31d-54cf-49b7-a4b7-ae4984221a56\") " pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.365178 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.365031    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1b11f31d-54cf-49b7-a4b7-ae4984221a56-default-certificate\") pod \"router-default-76b88fd7dd-5knhq\" (UID: \"1b11f31d-54cf-49b7-a4b7-ae4984221a56\") " pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.365178 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.365047    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5wdn\" (UniqueName: \"kubernetes.io/projected/1b11f31d-54cf-49b7-a4b7-ae4984221a56-kube-api-access-t5wdn\") pod \"router-default-76b88fd7dd-5knhq\" (UID: \"1b11f31d-54cf-49b7-a4b7-ae4984221a56\") " pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.365178 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.365081    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b11f31d-54cf-49b7-a4b7-ae4984221a56-service-ca-bundle\") pod \"router-default-76b88fd7dd-5knhq\" (UID: \"1b11f31d-54cf-49b7-a4b7-ae4984221a56\") " pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.365389 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.365243    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b11f31d-54cf-49b7-a4b7-ae4984221a56-metrics-certs\") pod \"router-default-76b88fd7dd-5knhq\" (UID: \"1b11f31d-54cf-49b7-a4b7-ae4984221a56\") " pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.365819 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.365788    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b11f31d-54cf-49b7-a4b7-ae4984221a56-service-ca-bundle\") pod \"router-default-76b88fd7dd-5knhq\" (UID: \"1b11f31d-54cf-49b7-a4b7-ae4984221a56\") " pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.367603 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.367573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b11f31d-54cf-49b7-a4b7-ae4984221a56-metrics-certs\") pod \"router-default-76b88fd7dd-5knhq\" (UID: \"1b11f31d-54cf-49b7-a4b7-ae4984221a56\") " pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.367722 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.367628    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1b11f31d-54cf-49b7-a4b7-ae4984221a56-stats-auth\") pod \"router-default-76b88fd7dd-5knhq\" (UID: \"1b11f31d-54cf-49b7-a4b7-ae4984221a56\") " pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.367996 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.367975    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1b11f31d-54cf-49b7-a4b7-ae4984221a56-default-certificate\") pod \"router-default-76b88fd7dd-5knhq\" (UID: \"1b11f31d-54cf-49b7-a4b7-ae4984221a56\") " pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.373769 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.373738    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6"
Sep 04 23:37:32.376708 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.376688    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92grx\" (UniqueName: \"kubernetes.io/projected/082f581e-6378-4981-bba8-fd66fa8923e1-kube-api-access-92grx\") pod \"downloads-85d98c998f-td8gm\" (UID: \"082f581e-6378-4981-bba8-fd66fa8923e1\") " pod="openshift-console/downloads-85d98c998f-td8gm"
Sep 04 23:37:32.377892 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.377875    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5wdn\" (UniqueName: \"kubernetes.io/projected/1b11f31d-54cf-49b7-a4b7-ae4984221a56-kube-api-access-t5wdn\") pod \"router-default-76b88fd7dd-5knhq\" (UID: \"1b11f31d-54cf-49b7-a4b7-ae4984221a56\") " pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.384734 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.384718    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-6fj75"
Sep 04 23:37:32.397940 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.397919    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9"
Sep 04 23:37:32.421181 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.421156    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"
Sep 04 23:37:32.457503 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.454175    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"
Sep 04 23:37:32.462059 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.461968    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c849d556-mcx9f"
Sep 04 23:37:32.506708 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.505014    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:32.511523 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.511416    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-td8gm"
Sep 04 23:37:32.677601 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.677577    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-vxcx6"]
Sep 04 23:37:32.679880 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:32.679849    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cc507b_07c5_4ced_bb71_2cbc7b2c4f00.slice/crio-099f0c87071d4389345728768d57be74c9e541f38cb3b95d78587db754aee3d6 WatchSource:0}: Error finding container 099f0c87071d4389345728768d57be74c9e541f38cb3b95d78587db754aee3d6: Status 404 returned error can't find the container with id 099f0c87071d4389345728768d57be74c9e541f38cb3b95d78587db754aee3d6
Sep 04 23:37:32.713024 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.713003    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-6fj75"]
Sep 04 23:37:32.715005 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:32.714979    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa30820_fda2_4e06_80a1_6e9de51833ea.slice/crio-8d8ecf3d31b011a213ce4f8323d17633969cad5ae98650ce379664b2175fd295 WatchSource:0}: Error finding container 8d8ecf3d31b011a213ce4f8323d17633969cad5ae98650ce379664b2175fd295: Status 404 returned error can't find the container with id 8d8ecf3d31b011a213ce4f8323d17633969cad5ae98650ce379664b2175fd295
Sep 04 23:37:32.747001 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.746947    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"]
Sep 04 23:37:32.749793 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:32.749769    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8215ba_ce6e_43a1_b1f6_bfb93ec67f66.slice/crio-ac1ab3318f1927049f32761732729d5c221e12063ca22b5e8530be655b0239dd WatchSource:0}: Error finding container ac1ab3318f1927049f32761732729d5c221e12063ca22b5e8530be655b0239dd: Status 404 returned error can't find the container with id ac1ab3318f1927049f32761732729d5c221e12063ca22b5e8530be655b0239dd
Sep 04 23:37:32.757850 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.757004    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9"]
Sep 04 23:37:32.760072 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:32.760052    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ef1223_76e2_4834_af84_b9ca9a19b579.slice/crio-ad3784a374937569ae7a664a61b137a222dc2f6fce703db819777700b9036d27 WatchSource:0}: Error finding container ad3784a374937569ae7a664a61b137a222dc2f6fce703db819777700b9036d27: Status 404 returned error can't find the container with id ad3784a374937569ae7a664a61b137a222dc2f6fce703db819777700b9036d27
Sep 04 23:37:32.786139 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.786114    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn"]
Sep 04 23:37:32.788716 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:32.788694    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fdc957b_b8aa_4522_b457_cf882a06bf3c.slice/crio-31a94b768b44baa174d168fe8f33983aeca5292a31572ac47b85102f1c87e52c WatchSource:0}: Error finding container 31a94b768b44baa174d168fe8f33983aeca5292a31572ac47b85102f1c87e52c: Status 404 returned error can't find the container with id 31a94b768b44baa174d168fe8f33983aeca5292a31572ac47b85102f1c87e52c
Sep 04 23:37:32.792100 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.791986    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c849d556-mcx9f"]
Sep 04 23:37:32.795480 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:32.795451    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda55b5a2c_cebb_4737_bbb3_afc50bd85e78.slice/crio-9215a9dddee48bf40ef2df6f55a22989007414bb72cea8d1f2c8e0893dee4bab WatchSource:0}: Error finding container 9215a9dddee48bf40ef2df6f55a22989007414bb72cea8d1f2c8e0893dee4bab: Status 404 returned error can't find the container with id 9215a9dddee48bf40ef2df6f55a22989007414bb72cea8d1f2c8e0893dee4bab
Sep 04 23:37:32.832043 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.832005    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76b88fd7dd-5knhq"]
Sep 04 23:37:32.835473 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:32.835452    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b11f31d_54cf_49b7_a4b7_ae4984221a56.slice/crio-cade82d561763a5c740f81f6a4be8d2afb354e4cfac5b3ce2249b1c9c2299f3c WatchSource:0}: Error finding container cade82d561763a5c740f81f6a4be8d2afb354e4cfac5b3ce2249b1c9c2299f3c: Status 404 returned error can't find the container with id cade82d561763a5c740f81f6a4be8d2afb354e4cfac5b3ce2249b1c9c2299f3c
Sep 04 23:37:32.852209 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:32.852149    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-td8gm"]
Sep 04 23:37:32.855899 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:32.855875    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082f581e_6378_4981_bba8_fd66fa8923e1.slice/crio-b78698c09fcbcc7e55b9efba89936cd5e285616912a8e076bf541612222f3e71 WatchSource:0}: Error finding container b78698c09fcbcc7e55b9efba89936cd5e285616912a8e076bf541612222f3e71: Status 404 returned error can't find the container with id b78698c09fcbcc7e55b9efba89936cd5e285616912a8e076bf541612222f3e71
Sep 04 23:37:33.478466 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.478431    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:33.483814 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.483782    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls\") pod \"image-registry-cdf99854b-t9dhs\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") " pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:33.680332 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.680289    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:33.698388 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.698292    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9" event={"ID":"08ef1223-76e2-4834-af84-b9ca9a19b579","Type":"ContainerStarted","Data":"ad3784a374937569ae7a664a61b137a222dc2f6fce703db819777700b9036d27"}
Sep 04 23:37:33.707982 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.707699    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f745f9b6-37b0-4710-a39f-282b4d388677-metrics-tls\") pod \"dns-default-rv66q\" (UID: \"f745f9b6-37b0-4710-a39f-282b4d388677\") " pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:33.714729 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.714666    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-td8gm" event={"ID":"082f581e-6378-4981-bba8-fd66fa8923e1","Type":"ContainerStarted","Data":"b78698c09fcbcc7e55b9efba89936cd5e285616912a8e076bf541612222f3e71"}
Sep 04 23:37:33.729150 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.728834    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6hx2z\""
Sep 04 23:37:33.733340 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.733266    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c849d556-mcx9f" event={"ID":"a55b5a2c-cebb-4737-bbb3-afc50bd85e78","Type":"ContainerStarted","Data":"9215a9dddee48bf40ef2df6f55a22989007414bb72cea8d1f2c8e0893dee4bab"}
Sep 04 23:37:33.737331 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.737003    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:33.747339 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.746207    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76b88fd7dd-5knhq" event={"ID":"1b11f31d-54cf-49b7-a4b7-ae4984221a56","Type":"ContainerStarted","Data":"de926681b2ed1284357fcae2fb8d90b12c066671004e027255702fba75f20b7d"}
Sep 04 23:37:33.747339 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.746264    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76b88fd7dd-5knhq" event={"ID":"1b11f31d-54cf-49b7-a4b7-ae4984221a56","Type":"ContainerStarted","Data":"cade82d561763a5c740f81f6a4be8d2afb354e4cfac5b3ce2249b1c9c2299f3c"}
Sep 04 23:37:33.767096 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.767037    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-6fj75" event={"ID":"3fa30820-fda2-4e06-80a1-6e9de51833ea","Type":"ContainerStarted","Data":"8d8ecf3d31b011a213ce4f8323d17633969cad5ae98650ce379664b2175fd295"}
Sep 04 23:37:33.768556 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.768455    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-76b88fd7dd-5knhq" podStartSLOduration=1.768416583 podStartE2EDuration="1.768416583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:33.763762825 +0000 UTC m=+66.010429904" watchObservedRunningTime="2026-09-04 23:37:33.768416583 +0000 UTC m=+66.015083660"
Sep 04 23:37:33.775229 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.775153    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q" event={"ID":"ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66","Type":"ContainerStarted","Data":"ac1ab3318f1927049f32761732729d5c221e12063ca22b5e8530be655b0239dd"}
Sep 04 23:37:33.779220 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.779137    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6" event={"ID":"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00","Type":"ContainerStarted","Data":"099f0c87071d4389345728768d57be74c9e541f38cb3b95d78587db754aee3d6"}
Sep 04 23:37:33.784215 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.784164    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn" event={"ID":"6fdc957b-b8aa-4522-b457-cf882a06bf3c","Type":"ContainerStarted","Data":"31a94b768b44baa174d168fe8f33983aeca5292a31572ac47b85102f1c87e52c"}
Sep 04 23:37:33.866761 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.866522    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4rhln\""
Sep 04 23:37:33.874589 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.874546    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:33.984251 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.983724    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs\") pod \"network-metrics-daemon-m8mxs\" (UID: \"38a67465-053c-4563-8c2c-8bff74b88a4b\") " pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:37:33.986698 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.986460    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 04 23:37:33.999888 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:33.999832    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a67465-053c-4563-8c2c-8bff74b88a4b-metrics-certs\") pod \"network-metrics-daemon-m8mxs\" (UID: \"38a67465-053c-4563-8c2c-8bff74b88a4b\") " pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:37:34.060614 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:34.059933    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cdf99854b-t9dhs"]
Sep 04 23:37:34.084905 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:34.084488    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:37:34.087066 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:34.086865    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 04 23:37:34.102050 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:34.101992    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7234a20-6651-4386-8521-484e928babba-original-pull-secret\") pod \"global-pull-secret-syncer-hq8l5\" (UID: \"e7234a20-6651-4386-8521-484e928babba\") " pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:37:34.199787 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:34.199749    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hq8l5"
Sep 04 23:37:34.208585 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:34.208563    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8jjtb\""
Sep 04 23:37:34.216633 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:34.216610    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8mxs"
Sep 04 23:37:34.507425 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:34.507385    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:34.510848 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:34.510824    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:34.801817 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:34.801734    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:34.809046 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:34.809019    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-76b88fd7dd-5knhq"
Sep 04 23:37:35.374423 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:35.373681    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m8mxs"]
Sep 04 23:37:35.380487 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:35.380373    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rv66q"]
Sep 04 23:37:35.454916 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:35.451688    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hq8l5"]
Sep 04 23:37:35.818423 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:35.817461    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cdf99854b-t9dhs" event={"ID":"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6","Type":"ContainerStarted","Data":"e7ef1ba6a46916f136dd7480a611614706a1a8a74c1ad183da962cb04eae1a8e"}
Sep 04 23:37:35.818423 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:35.817526    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cdf99854b-t9dhs" event={"ID":"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6","Type":"ContainerStarted","Data":"f5b374751c352aff4017ae19cbe64948a956b148cc47edd5c221977f51902dee"}
Sep 04 23:37:35.818423 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:35.818372    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:35.829646 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:35.828318    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-6fj75" event={"ID":"3fa30820-fda2-4e06-80a1-6e9de51833ea","Type":"ContainerStarted","Data":"47ac4376d6994dfebfa280f37aafad2c4c2f1d40458fb6015d9e68db22eea4d9"}
Sep 04 23:37:35.839833 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:35.837915    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cdf99854b-t9dhs" podStartSLOduration=67.837892169 podStartE2EDuration="1m7.837892169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:35.836392192 +0000 UTC m=+68.083059298" watchObservedRunningTime="2026-09-04 23:37:35.837892169 +0000 UTC m=+68.084559248"
Sep 04 23:37:35.852638 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:35.852475    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-6fj75" podStartSLOduration=1.436059801 podStartE2EDuration="3.852453084s" totalImagesPullingTime="2.416393283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:35.850775061 +0000 UTC m=+68.097442144" watchObservedRunningTime="2026-09-04 23:37:35.852453084 +0000 UTC m=+68.099120163"
Sep 04 23:37:36.550736 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:36.550691    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7234a20_6651_4386_8521_484e928babba.slice/crio-8c9e04c6740476b6bb5554447d25628b64df6d67407f7f91c8426433e8fc9139 WatchSource:0}: Error finding container 8c9e04c6740476b6bb5554447d25628b64df6d67407f7f91c8426433e8fc9139: Status 404 returned error can't find the container with id 8c9e04c6740476b6bb5554447d25628b64df6d67407f7f91c8426433e8fc9139
Sep 04 23:37:36.835606 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:36.835467    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m8mxs" event={"ID":"38a67465-053c-4563-8c2c-8bff74b88a4b","Type":"ContainerStarted","Data":"dc93eea37a51406c382b767c885a054444ac4bdf69f0c452bed42179913381f3"}
Sep 04 23:37:36.836944 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:36.836919    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hq8l5" event={"ID":"e7234a20-6651-4386-8521-484e928babba","Type":"ContainerStarted","Data":"8c9e04c6740476b6bb5554447d25628b64df6d67407f7f91c8426433e8fc9139"}
Sep 04 23:37:37.598842 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:37:37.598801    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf745f9b6_37b0_4710_a39f_282b4d388677.slice/crio-028f38c95c08a20558185d3c6df67f8dc1edad2648c9fd33c60516db89dab7e2 WatchSource:0}: Error finding container 028f38c95c08a20558185d3c6df67f8dc1edad2648c9fd33c60516db89dab7e2: Status 404 returned error can't find the container with id 028f38c95c08a20558185d3c6df67f8dc1edad2648c9fd33c60516db89dab7e2
Sep 04 23:37:37.842532 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:37.842491    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rv66q" event={"ID":"f745f9b6-37b0-4710-a39f-282b4d388677","Type":"ContainerStarted","Data":"028f38c95c08a20558185d3c6df67f8dc1edad2648c9fd33c60516db89dab7e2"}
Sep 04 23:37:41.602126 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:41.601815    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mgg4d"
Sep 04 23:37:53.746176 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.745932    2562 patch_prober.go:28] interesting pod/image-registry-cdf99854b-t9dhs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 04 23:37:53.746176 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.745985    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-cdf99854b-t9dhs" podUID="ceeabe27-d2d7-4819-aa74-cc9609f8e3e6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 23:37:53.916746 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.916706    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hq8l5" event={"ID":"e7234a20-6651-4386-8521-484e928babba","Type":"ContainerStarted","Data":"4a1a717a27730a49666fe58b3d539af75275db0281afcad3acad3fef930dfcf9"}
Sep 04 23:37:53.919450 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.919391    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-td8gm" event={"ID":"082f581e-6378-4981-bba8-fd66fa8923e1","Type":"ContainerStarted","Data":"0eeff38235281be79a2f7a5eacf9f5a9415f143f80d01cd6dc78496764d45090"}
Sep 04 23:37:53.919959 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.919911    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-td8gm"
Sep 04 23:37:53.922009 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.921913    2562 patch_prober.go:28] interesting pod/downloads-85d98c998f-td8gm container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.24:8080/\": dial tcp 10.132.0.24:8080: connect: connection refused" start-of-body=
Sep 04 23:37:53.922009 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.921960    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-td8gm" podUID="082f581e-6378-4981-bba8-fd66fa8923e1" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.24:8080/\": dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 04 23:37:53.922530 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.922494    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c849d556-mcx9f" event={"ID":"a55b5a2c-cebb-4737-bbb3-afc50bd85e78","Type":"ContainerStarted","Data":"bc5efbd192f397375b2b597e9d9c685e0b0032dfa963e4ad8d00c091b65765ec"}
Sep 04 23:37:53.924360 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.924335    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q" event={"ID":"ed8215ba-ce6e-43a1-b1f6-bfb93ec67f66","Type":"ContainerStarted","Data":"14d237938adc9ca33350d212aac3b77d056bfe7f139c6d683c844b2f192becfd"}
Sep 04 23:37:53.924863 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.924789    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"
Sep 04 23:37:53.926464 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.926441    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q"
Sep 04 23:37:53.926702 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.926624    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6" event={"ID":"b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00","Type":"ContainerStarted","Data":"5055004d8f3543a74b0c1b2a9276f910db31bf503681e424e6444e508d8ac6dc"}
Sep 04 23:37:53.929877 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.929853    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn" event={"ID":"6fdc957b-b8aa-4522-b457-cf882a06bf3c","Type":"ContainerStarted","Data":"32f2f079e668487be9b8e7414ea7ffa3c367f5edc858ea632e902e30880012c4"}
Sep 04 23:37:53.929976 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.929887    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn" event={"ID":"6fdc957b-b8aa-4522-b457-cf882a06bf3c","Type":"ContainerStarted","Data":"8f24721323d09695d259f8b48ce7e05012dd256794bc030201a253aabcce87ff"}
Sep 04 23:37:53.929976 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.929904    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn" event={"ID":"6fdc957b-b8aa-4522-b457-cf882a06bf3c","Type":"ContainerStarted","Data":"b65c0032e5f99168e2fa0ded6d29d8f117feb57130158801ea8ae8b591e23e3b"}
Sep 04 23:37:53.932930 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.932905    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rv66q" event={"ID":"f745f9b6-37b0-4710-a39f-282b4d388677","Type":"ContainerStarted","Data":"b72545b2b4e940232f4b18cac48be42810719f6dfdd25b16fe21313607612471"}
Sep 04 23:37:53.933024 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.932953    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rv66q" event={"ID":"f745f9b6-37b0-4710-a39f-282b4d388677","Type":"ContainerStarted","Data":"9ae7b12b61a86244c8c0ecb46c127b1c36748e46c3f2d29b4177e6186fa16635"}
Sep 04 23:37:53.933427 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.933408    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rv66q"
Sep 04 23:37:53.935215 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.935130    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hq8l5" podStartSLOduration=69.364062394 podStartE2EDuration="1m25.935113719s" totalImagesPullingTime="16.571051325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:53.934365528 +0000 UTC m=+86.181032607" watchObservedRunningTime="2026-09-04 23:37:53.935113719 +0000 UTC m=+86.181780799"
Sep 04 23:37:53.936319 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.936261    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9" event={"ID":"08ef1223-76e2-4834-af84-b9ca9a19b579","Type":"ContainerStarted","Data":"853f46eff3f806e702b1cced05134d6149d1feedfdf3a8d3c358dd31f06ed709"}
Sep 04 23:37:53.940071 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.940045    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m8mxs" event={"ID":"38a67465-053c-4563-8c2c-8bff74b88a4b","Type":"ContainerStarted","Data":"ad5f5ff413f72cc2e4e46f6142b96367830c6fe4ee8ff5be5ae38ae53908a9a1"}
Sep 04 23:37:53.953823 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.953768    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-vxcx6" podStartSLOduration=8.79251748 podStartE2EDuration="21.953731184s" totalImagesPullingTime="13.161213704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:53.952367094 +0000 UTC m=+86.199034165" watchObservedRunningTime="2026-09-04 23:37:53.953731184 +0000 UTC m=+86.200398264"
Sep 04 23:37:53.987014 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:53.984322    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c849d556-mcx9f" podStartSLOduration=1.720013299 podStartE2EDuration="21.984282914s" totalImagesPullingTime="20.264269615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:53.967697493 +0000 UTC m=+86.214364575" watchObservedRunningTime="2026-09-04 23:37:53.984282914 +0000 UTC m=+86.230949995"
Sep 04 23:37:54.007678 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:54.006697    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rv66q" podStartSLOduration=37.481626668 podStartE2EDuration="53.006673539s" totalImagesPullingTime="15.525046871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:53.987056086 +0000 UTC m=+86.233723165" watchObservedRunningTime="2026-09-04 23:37:54.006673539 +0000 UTC m=+86.253340611"
Sep 04 23:37:54.023716 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:54.023454    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8f556f98-8dd7q" podStartSLOduration=1.714063485 podStartE2EDuration="22.023420294s" totalImagesPullingTime="20.309356809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:54.022424426 +0000 UTC m=+86.269091509" watchObservedRunningTime="2026-09-04 23:37:54.023420294 +0000 UTC m=+86.270087375"
Sep 04 23:37:54.024569 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:54.024250    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c969d7d87-5k6qn" podStartSLOduration=1.689686456 podStartE2EDuration="22.024234582s" totalImagesPullingTime="20.334548126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:54.007327434 +0000 UTC m=+86.253994519" watchObservedRunningTime="2026-09-04 23:37:54.024234582 +0000 UTC m=+86.270901662"
Sep 04 23:37:54.042715 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:54.042646    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-td8gm" podStartSLOduration=1.384139499 podStartE2EDuration="22.04262715s" totalImagesPullingTime="20.658487651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:54.041948599 +0000 UTC m=+86.288615680" watchObservedRunningTime="2026-09-04 23:37:54.04262715 +0000 UTC m=+86.289294230"
Sep 04 23:37:54.058723 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:54.058619    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vd6n9" podStartSLOduration=11.443351136 podStartE2EDuration="22.058585988s" totalImagesPullingTime="10.615234852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:54.056809447 +0000 UTC m=+86.303476527" watchObservedRunningTime="2026-09-04 23:37:54.058585988 +0000 UTC m=+86.305253068"
Sep 04 23:37:54.878319 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:54.878265    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m8mxs" podStartSLOduration=70.304220109 podStartE2EDuration="1m26.878238705s" totalImagesPullingTime="16.574018596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:54.072884094 +0000 UTC m=+86.319551176" watchObservedRunningTime="2026-09-04 23:37:54.878238705 +0000 UTC m=+87.124905785"
Sep 04 23:37:54.879683 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:54.879642    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8xjqb"]
Sep 04 23:37:54.913705 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:54.913641    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8xjqb"]
Sep 04 23:37:54.913854 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:54.913781    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:54.916625 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:54.916362    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 04 23:37:54.916625 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:54.916376    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qcf5d\""
Sep 04 23:37:54.916625 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:54.916363    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 04 23:37:54.949219 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:54.949185    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m8mxs" event={"ID":"38a67465-053c-4563-8c2c-8bff74b88a4b","Type":"ContainerStarted","Data":"96324290d077b1fce965b1defc1c904f3185f63c3840194f4d238dd2b4431591"}
Sep 04 23:37:54.954896 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:54.954867    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-td8gm"
Sep 04 23:37:55.078343 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:55.078300    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b41b8374-a266-47db-9ee5-13916d81025a-crio-socket\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:55.078542 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:55.078368    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:55.078775 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:55.078709    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:55.079363 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:55.079173    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b41b8374-a266-47db-9ee5-13916d81025a-data-volume\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:55.080538 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:55.080508    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r62mc\" (UniqueName: \"kubernetes.io/projected/b41b8374-a266-47db-9ee5-13916d81025a-kube-api-access-r62mc\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:55.181966 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:55.181866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:55.182192 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:55.182036    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 23:37:55.182192 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:55.182055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b41b8374-a266-47db-9ee5-13916d81025a-data-volume\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:55.182192 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:55.182129    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm podName:b41b8374-a266-47db-9ee5-13916d81025a nodeName:}" failed. No retries permitted until 2026-09-04 23:37:55.682105336 +0000 UTC m=+87.928772404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8xjqb" (UID: "b41b8374-a266-47db-9ee5-13916d81025a") : configmap "kube-rbac-proxy" not found
Sep 04 23:37:55.182192 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:55.182155    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r62mc\" (UniqueName: \"kubernetes.io/projected/b41b8374-a266-47db-9ee5-13916d81025a-kube-api-access-r62mc\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:55.182415 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:55.182224    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b41b8374-a266-47db-9ee5-13916d81025a-crio-socket\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:55.182415 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:55.182260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:55.182415 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:55.182350    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b41b8374-a266-47db-9ee5-13916d81025a-crio-socket\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:55.182415 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:55.182368    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:55.182415 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:55.182406    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls podName:b41b8374-a266-47db-9ee5-13916d81025a nodeName:}" failed. No retries permitted until 2026-09-04 23:37:55.682395069 +0000 UTC m=+87.929062126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xjqb" (UID: "b41b8374-a266-47db-9ee5-13916d81025a") : secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:55.182687 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:55.182420    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b41b8374-a266-47db-9ee5-13916d81025a-data-volume\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:55.190689 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:55.190643    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r62mc\" (UniqueName: \"kubernetes.io/projected/b41b8374-a266-47db-9ee5-13916d81025a-kube-api-access-r62mc\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:55.687555 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:55.687512    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:55.687839 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:55.687812    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:55.687934 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:55.687697    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:55.688027 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:55.687903    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 23:37:55.688027 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:55.688028    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls podName:b41b8374-a266-47db-9ee5-13916d81025a nodeName:}" failed. No retries permitted until 2026-09-04 23:37:56.688002753 +0000 UTC m=+88.934669828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xjqb" (UID: "b41b8374-a266-47db-9ee5-13916d81025a") : secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:55.688219 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:55.688125    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm podName:b41b8374-a266-47db-9ee5-13916d81025a nodeName:}" failed. No retries permitted until 2026-09-04 23:37:56.688105012 +0000 UTC m=+88.934772092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8xjqb" (UID: "b41b8374-a266-47db-9ee5-13916d81025a") : configmap "kube-rbac-proxy" not found
Sep 04 23:37:56.637300 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:56.637260    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-6fj75_3fa30820-fda2-4e06-80a1-6e9de51833ea/networking-console-plugin/0.log"
Sep 04 23:37:56.697707 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:56.697664    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:56.697876 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:56.697742    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:56.697876 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:56.697768    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:56.697876 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:56.697835    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls podName:b41b8374-a266-47db-9ee5-13916d81025a nodeName:}" failed. No retries permitted until 2026-09-04 23:37:58.697817419 +0000 UTC m=+90.944484476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xjqb" (UID: "b41b8374-a266-47db-9ee5-13916d81025a") : secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:56.698049 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:56.697872    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 23:37:56.698049 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:56.697931    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm podName:b41b8374-a266-47db-9ee5-13916d81025a nodeName:}" failed. No retries permitted until 2026-09-04 23:37:58.6979187 +0000 UTC m=+90.944585757 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8xjqb" (UID: "b41b8374-a266-47db-9ee5-13916d81025a") : configmap "kube-rbac-proxy" not found
Sep 04 23:37:56.837217 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:56.837178    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 04 23:37:57.040353 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:57.040308    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/2.log"
Sep 04 23:37:57.639455 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:57.639418    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-td8gm_082f581e-6378-4981-bba8-fd66fa8923e1/download-server/0.log"
Sep 04 23:37:57.848314 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:57.848281    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:37:58.636917 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:58.636886    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rv66q_f745f9b6-37b0-4710-a39f-282b4d388677/dns/0.log"
Sep 04 23:37:58.716207 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:58.716168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:58.716709 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:58.716290    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:37:58.716709 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:58.716306    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 23:37:58.716709 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:58.716401    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm podName:b41b8374-a266-47db-9ee5-13916d81025a nodeName:}" failed. No retries permitted until 2026-09-04 23:38:02.716378259 +0000 UTC m=+94.963045322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8xjqb" (UID: "b41b8374-a266-47db-9ee5-13916d81025a") : configmap "kube-rbac-proxy" not found
Sep 04 23:37:58.716709 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:58.716426    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:58.716709 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:37:58.716479    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls podName:b41b8374-a266-47db-9ee5-13916d81025a nodeName:}" failed. No retries permitted until 2026-09-04 23:38:02.716463492 +0000 UTC m=+94.963130548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xjqb" (UID: "b41b8374-a266-47db-9ee5-13916d81025a") : secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:58.836483 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:58.836451    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rv66q_f745f9b6-37b0-4710-a39f-282b4d388677/kube-rbac-proxy/0.log"
Sep 04 23:37:59.325393 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:59.325352    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cdf99854b-t9dhs"]
Sep 04 23:37:59.436886 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:59.436854    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dpw8x_f507b59a-26e6-4c41-b0ea-5a428a8adba8/dns-node-resolver/0.log"
Sep 04 23:37:59.836599 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:37:59.836570    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-cdf99854b-t9dhs_ceeabe27-d2d7-4819-aa74-cc9609f8e3e6/registry/0.log"
Sep 04 23:38:00.238331 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:00.238295    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pjgpk_0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9/node-ca/0.log"
Sep 04 23:38:00.637087 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:00.637014    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76b88fd7dd-5knhq_1b11f31d-54cf-49b7-a4b7-ae4984221a56/router/0.log"
Sep 04 23:38:00.836453 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:00.836423    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fmc2z_3d903250-92e3-48c2-b52f-e70ff30805a0/serve-healthcheck-canary/0.log"
Sep 04 23:38:02.749491 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:02.749452    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:38:02.749909 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:02.749522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:38:02.749909 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:38:02.749577    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 23:38:02.749909 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:38:02.749609    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 23:38:02.749909 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:38:02.749685    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm podName:b41b8374-a266-47db-9ee5-13916d81025a nodeName:}" failed. No retries permitted until 2026-09-04 23:38:10.749642559 +0000 UTC m=+102.996309619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8xjqb" (UID: "b41b8374-a266-47db-9ee5-13916d81025a") : configmap "kube-rbac-proxy" not found
Sep 04 23:38:02.749909 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:38:02.749707    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls podName:b41b8374-a266-47db-9ee5-13916d81025a nodeName:}" failed. No retries permitted until 2026-09-04 23:38:10.749697143 +0000 UTC m=+102.996364201 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xjqb" (UID: "b41b8374-a266-47db-9ee5-13916d81025a") : secret "insights-runtime-extractor-tls" not found
Sep 04 23:38:04.956981 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:04.956951    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rv66q"
Sep 04 23:38:06.063502 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.063463    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5b8gj"]
Sep 04 23:38:06.089603 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.089566    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.092160 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.092088    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 04 23:38:06.092296 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.092268    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 04 23:38:06.093315 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.093153    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 04 23:38:06.093315 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.093218    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pq79h\""
Sep 04 23:38:06.093315 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.093245    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 04 23:38:06.181047 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.181010    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-tls\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.181230 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.181054    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc21b587-b979-4a80-bd88-7e7b40dbacad-metrics-client-ca\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.181230 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.181154    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck88b\" (UniqueName: \"kubernetes.io/projected/dc21b587-b979-4a80-bd88-7e7b40dbacad-kube-api-access-ck88b\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.181230 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.181202    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.181392 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.181240    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc21b587-b979-4a80-bd88-7e7b40dbacad-sys\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.181392 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.181261    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc21b587-b979-4a80-bd88-7e7b40dbacad-root\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.181392 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.181291    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-accelerators-collector-config\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.181392 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.181319    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-textfile\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.181392 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.181360    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-wtmp\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.282124 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.282090    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-tls\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.282351 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.282336    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc21b587-b979-4a80-bd88-7e7b40dbacad-metrics-client-ca\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.282469 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.282457    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck88b\" (UniqueName: \"kubernetes.io/projected/dc21b587-b979-4a80-bd88-7e7b40dbacad-kube-api-access-ck88b\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.282970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.282947    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.283141 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.283127    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc21b587-b979-4a80-bd88-7e7b40dbacad-sys\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.283303 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.283290    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc21b587-b979-4a80-bd88-7e7b40dbacad-root\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.283404 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.283380    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc21b587-b979-4a80-bd88-7e7b40dbacad-root\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.283459 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.283263    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc21b587-b979-4a80-bd88-7e7b40dbacad-sys\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.283459 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:38:06.282248    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 04 23:38:06.283556 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:38:06.283482    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-tls podName:dc21b587-b979-4a80-bd88-7e7b40dbacad nodeName:}" failed. No retries permitted until 2026-09-04 23:38:06.78346296 +0000 UTC m=+99.030130022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-tls") pod "node-exporter-5b8gj" (UID: "dc21b587-b979-4a80-bd88-7e7b40dbacad") : secret "node-exporter-tls" not found
Sep 04 23:38:06.283627 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.283395    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-accelerators-collector-config\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.283750 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.283002    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc21b587-b979-4a80-bd88-7e7b40dbacad-metrics-client-ca\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.283873 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.283859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-textfile\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.284337 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.284321    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-wtmp\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.284680 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.284241    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-textfile\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.284804 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.284544    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-wtmp\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.284903 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.284225    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-accelerators-collector-config\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.287041 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.286935    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.292009 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.291988    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck88b\" (UniqueName: \"kubernetes.io/projected/dc21b587-b979-4a80-bd88-7e7b40dbacad-kube-api-access-ck88b\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.789415 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.789381    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-tls\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:06.792108 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:06.792081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc21b587-b979-4a80-bd88-7e7b40dbacad-node-exporter-tls\") pod \"node-exporter-5b8gj\" (UID: \"dc21b587-b979-4a80-bd88-7e7b40dbacad\") " pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:07.000345 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:07.000311    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5b8gj"
Sep 04 23:38:07.045474 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:38:07.045441    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc21b587_b979_4a80_bd88_7e7b40dbacad.slice/crio-8b4595762e324c17956c9c13a2abbc3386f605fb659a585726fe9ffa3d2e3aac WatchSource:0}: Error finding container 8b4595762e324c17956c9c13a2abbc3386f605fb659a585726fe9ffa3d2e3aac: Status 404 returned error can't find the container with id 8b4595762e324c17956c9c13a2abbc3386f605fb659a585726fe9ffa3d2e3aac
Sep 04 23:38:08.010491 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:08.010453    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5b8gj" event={"ID":"dc21b587-b979-4a80-bd88-7e7b40dbacad","Type":"ContainerStarted","Data":"8b4595762e324c17956c9c13a2abbc3386f605fb659a585726fe9ffa3d2e3aac"}
Sep 04 23:38:09.016122 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:09.016083    2562 generic.go:356] "Generic (PLEG): container finished" podID="dc21b587-b979-4a80-bd88-7e7b40dbacad" containerID="a742ad89bc572aaf28f7e5957ab3c65f58156507faa0a922f2ead69c3cb937f5" exitCode=0
Sep 04 23:38:09.016512 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:09.016181    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5b8gj" event={"ID":"dc21b587-b979-4a80-bd88-7e7b40dbacad","Type":"ContainerDied","Data":"a742ad89bc572aaf28f7e5957ab3c65f58156507faa0a922f2ead69c3cb937f5"}
Sep 04 23:38:10.022638 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:10.022599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5b8gj" event={"ID":"dc21b587-b979-4a80-bd88-7e7b40dbacad","Type":"ContainerStarted","Data":"79d7ab091195327b9207fdc346b016038be32b29321ad5a3d1cb9850316d904a"}
Sep 04 23:38:10.023130 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:10.022689    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5b8gj" event={"ID":"dc21b587-b979-4a80-bd88-7e7b40dbacad","Type":"ContainerStarted","Data":"6cf9f98f8bf2ccb44b453e75146ee9b794c893954b058cab1a838aca3ddf8993"}
Sep 04 23:38:10.041641 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:10.041594    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5b8gj" podStartSLOduration=2.380875385 podStartE2EDuration="4.041577665s" totalImagesPullingTime="1.565892319s" totalInitContainerRuntime="94.809961ms" isStatefulPod=true podCreationTimestamp="2026-09-04 23:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:38:10.03961128 +0000 UTC m=+102.286278359" watchObservedRunningTime="2026-09-04 23:38:10.041577665 +0000 UTC m=+102.288244783"
Sep 04 23:38:10.829410 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:10.829374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:38:10.829591 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:10.829433    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:38:10.829955 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:10.829931    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b41b8374-a266-47db-9ee5-13916d81025a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:38:10.832057 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:10.832034    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b41b8374-a266-47db-9ee5-13916d81025a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xjqb\" (UID: \"b41b8374-a266-47db-9ee5-13916d81025a\") " pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:38:11.127760 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:11.127683    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8xjqb"
Sep 04 23:38:11.438315 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:11.438233    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8xjqb"]
Sep 04 23:38:11.441809 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:38:11.441781    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41b8374_a266_47db_9ee5_13916d81025a.slice/crio-ce09119eb90247741ff750f954696d83910ee8a0cfad2da53073c5bfe9e46b14 WatchSource:0}: Error finding container ce09119eb90247741ff750f954696d83910ee8a0cfad2da53073c5bfe9e46b14: Status 404 returned error can't find the container with id ce09119eb90247741ff750f954696d83910ee8a0cfad2da53073c5bfe9e46b14
Sep 04 23:38:12.033306 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:12.033264    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8xjqb" event={"ID":"b41b8374-a266-47db-9ee5-13916d81025a","Type":"ContainerStarted","Data":"ffb1a684e2a02a69e478cb94770d2a54fad714bd585bbe2b43d184d18f5e1b6b"}
Sep 04 23:38:12.033583 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:12.033328    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8xjqb" event={"ID":"b41b8374-a266-47db-9ee5-13916d81025a","Type":"ContainerStarted","Data":"ce09119eb90247741ff750f954696d83910ee8a0cfad2da53073c5bfe9e46b14"}
Sep 04 23:38:13.039417 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:13.039378    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8xjqb" event={"ID":"b41b8374-a266-47db-9ee5-13916d81025a","Type":"ContainerStarted","Data":"7a09a53a4a6cdc1d61322c495263692d2c8db9079e9c91261f6b1737b9c41f67"}
Sep 04 23:38:15.049444 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:15.049406    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8xjqb" event={"ID":"b41b8374-a266-47db-9ee5-13916d81025a","Type":"ContainerStarted","Data":"52b8f4823d020c82c9fe60c9d8f881b4c68c2d52fd7cbd00b26cf6984ac007e4"}
Sep 04 23:38:15.066144 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:15.066090    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8xjqb" podStartSLOduration=17.949505373 podStartE2EDuration="21.066074841s" totalImagesPullingTime="3.116569468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:37:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:38:15.065129892 +0000 UTC m=+107.311796970" watchObservedRunningTime="2026-09-04 23:38:15.066074841 +0000 UTC m=+107.312741927"
Sep 04 23:38:24.353596 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.353531    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-cdf99854b-t9dhs" podUID="ceeabe27-d2d7-4819-aa74-cc9609f8e3e6" containerName="registry" containerID="cri-o://e7ef1ba6a46916f136dd7480a611614706a1a8a74c1ad183da962cb04eae1a8e" gracePeriod=30
Sep 04 23:38:24.631104 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.631025    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:38:24.744075 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.744035    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls\") pod \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") "
Sep 04 23:38:24.744249 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.744091    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-ca-trust-extracted\") pod \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") "
Sep 04 23:38:24.744249 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.744123    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-certificates\") pod \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") "
Sep 04 23:38:24.744249 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.744154    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-installation-pull-secrets\") pod \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") "
Sep 04 23:38:24.744249 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.744183    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-bound-sa-token\") pod \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") "
Sep 04 23:38:24.744249 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.744207    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-image-registry-private-configuration\") pod \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") "
Sep 04 23:38:24.744493 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.744254    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-trusted-ca\") pod \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") "
Sep 04 23:38:24.744493 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.744297    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4ll6\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-kube-api-access-w4ll6\") pod \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\" (UID: \"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6\") "
Sep 04 23:38:24.744776 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.744696    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6" (UID: "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:38:24.744909 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.744775    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6" (UID: "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:38:24.746725 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.746676    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6" (UID: "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:38:24.746900 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.746875    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6" (UID: "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:38:24.746999 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.746975    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6" (UID: "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:38:24.747039 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.746993    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6" (UID: "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:38:24.747074 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.747048    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-kube-api-access-w4ll6" (OuterVolumeSpecName: "kube-api-access-w4ll6") pod "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6" (UID: "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6"). InnerVolumeSpecName "kube-api-access-w4ll6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:38:24.752247 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.752216    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6" (UID: "ceeabe27-d2d7-4819-aa74-cc9609f8e3e6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:38:24.845411 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.845372    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-installation-pull-secrets\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:38:24.845411 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.845403    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-bound-sa-token\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:38:24.845612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.845420    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-image-registry-private-configuration\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:38:24.845612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.845433    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-trusted-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:38:24.845612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.845445    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4ll6\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-kube-api-access-w4ll6\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:38:24.845612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.845455    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:38:24.845612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.845466    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-ca-trust-extracted\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:38:24.845612 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:24.845477    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6-registry-certificates\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:38:25.092186 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:25.092154    2562 generic.go:356] "Generic (PLEG): container finished" podID="ceeabe27-d2d7-4819-aa74-cc9609f8e3e6" containerID="e7ef1ba6a46916f136dd7480a611614706a1a8a74c1ad183da962cb04eae1a8e" exitCode=0
Sep 04 23:38:25.092351 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:25.092201    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cdf99854b-t9dhs"
Sep 04 23:38:25.092351 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:25.092222    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cdf99854b-t9dhs" event={"ID":"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6","Type":"ContainerDied","Data":"e7ef1ba6a46916f136dd7480a611614706a1a8a74c1ad183da962cb04eae1a8e"}
Sep 04 23:38:25.092351 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:25.092262    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cdf99854b-t9dhs" event={"ID":"ceeabe27-d2d7-4819-aa74-cc9609f8e3e6","Type":"ContainerDied","Data":"f5b374751c352aff4017ae19cbe64948a956b148cc47edd5c221977f51902dee"}
Sep 04 23:38:25.092351 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:25.092283    2562 scope.go:117] "RemoveContainer" containerID="e7ef1ba6a46916f136dd7480a611614706a1a8a74c1ad183da962cb04eae1a8e"
Sep 04 23:38:25.101453 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:25.101433    2562 scope.go:117] "RemoveContainer" containerID="e7ef1ba6a46916f136dd7480a611614706a1a8a74c1ad183da962cb04eae1a8e"
Sep 04 23:38:25.101783 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:38:25.101756    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ef1ba6a46916f136dd7480a611614706a1a8a74c1ad183da962cb04eae1a8e\": container with ID starting with e7ef1ba6a46916f136dd7480a611614706a1a8a74c1ad183da962cb04eae1a8e not found: ID does not exist" containerID="e7ef1ba6a46916f136dd7480a611614706a1a8a74c1ad183da962cb04eae1a8e"
Sep 04 23:38:25.101838 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:25.101793    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ef1ba6a46916f136dd7480a611614706a1a8a74c1ad183da962cb04eae1a8e"} err="failed to get container status \"e7ef1ba6a46916f136dd7480a611614706a1a8a74c1ad183da962cb04eae1a8e\": rpc error: code = NotFound desc = could not find container \"e7ef1ba6a46916f136dd7480a611614706a1a8a74c1ad183da962cb04eae1a8e\": container with ID starting with e7ef1ba6a46916f136dd7480a611614706a1a8a74c1ad183da962cb04eae1a8e not found: ID does not exist"
Sep 04 23:38:25.113445 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:25.113425    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cdf99854b-t9dhs"]
Sep 04 23:38:25.116800 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:25.116778    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cdf99854b-t9dhs"]
Sep 04 23:38:26.359844 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:26.359811    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceeabe27-d2d7-4819-aa74-cc9609f8e3e6" path="/var/lib/kubelet/pods/ceeabe27-d2d7-4819-aa74-cc9609f8e3e6/volumes"
Sep 04 23:38:41.156960 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:41.156925    2562 generic.go:356] "Generic (PLEG): container finished" podID="91cf8c41-30cb-4ce3-a9b1-193bdce73a60" containerID="12fcb68aade414abc5617266689146f2c78e3d796f3363c5a99befc23f37f91d" exitCode=0
Sep 04 23:38:41.157460 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:41.157001    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp" event={"ID":"91cf8c41-30cb-4ce3-a9b1-193bdce73a60","Type":"ContainerDied","Data":"12fcb68aade414abc5617266689146f2c78e3d796f3363c5a99befc23f37f91d"}
Sep 04 23:38:41.157460 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:41.157426    2562 scope.go:117] "RemoveContainer" containerID="12fcb68aade414abc5617266689146f2c78e3d796f3363c5a99befc23f37f91d"
Sep 04 23:38:41.158672 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:41.158633    2562 generic.go:356] "Generic (PLEG): container finished" podID="2b6f2cdb-29a1-4245-83c3-1bf5eb182073" containerID="ee7eb28243b322574a4d18a0294036de0dcb97e4c11e8830acf1a32f746c60ce" exitCode=0
Sep 04 23:38:41.158736 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:41.158686    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687" event={"ID":"2b6f2cdb-29a1-4245-83c3-1bf5eb182073","Type":"ContainerDied","Data":"ee7eb28243b322574a4d18a0294036de0dcb97e4c11e8830acf1a32f746c60ce"}
Sep 04 23:38:41.158954 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:41.158941    2562 scope.go:117] "RemoveContainer" containerID="ee7eb28243b322574a4d18a0294036de0dcb97e4c11e8830acf1a32f746c60ce"
Sep 04 23:38:42.165024 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:42.164989    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fbpjp" event={"ID":"91cf8c41-30cb-4ce3-a9b1-193bdce73a60","Type":"ContainerStarted","Data":"84b6df7dfe03a7adf2e90f13f0716b4477e41cc6d19bf3110f451977f6c24a36"}
Sep 04 23:38:42.166833 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:38:42.166808    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bz687" event={"ID":"2b6f2cdb-29a1-4245-83c3-1bf5eb182073","Type":"ContainerStarted","Data":"c8eda63fb0bb1329f0b24cc409bbfd83a7b6d92a3303854b37130dce869a9f9d"}
Sep 04 23:41:28.170964 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:28.170934    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 04 23:41:28.174426 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:28.174402    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 04 23:41:28.174933 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:28.174916    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 04 23:41:28.178720 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:28.178701    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 04 23:41:28.182549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:28.182531    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 04 23:41:36.454669 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.454628    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"]
Sep 04 23:41:36.456908 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.454942    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceeabe27-d2d7-4819-aa74-cc9609f8e3e6" containerName="registry"
Sep 04 23:41:36.456908 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.454954    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceeabe27-d2d7-4819-aa74-cc9609f8e3e6" containerName="registry"
Sep 04 23:41:36.456908 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.455008    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceeabe27-d2d7-4819-aa74-cc9609f8e3e6" containerName="registry"
Sep 04 23:41:36.457818 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.457803    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.460577 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.460553    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 04 23:41:36.460577 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.460573    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 04 23:41:36.460950 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.460935    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 04 23:41:36.461034 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.461016    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mm9nl\""
Sep 04 23:41:36.461082 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.461065    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 04 23:41:36.464379 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.464355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25v4m\" (UniqueName: \"kubernetes.io/projected/f1c5f4cf-5b1a-4aad-b528-592dcb82370f-kube-api-access-25v4m\") pod \"opendatahub-operator-controller-manager-6b9496bc86-2xc4x\" (UID: \"f1c5f4cf-5b1a-4aad-b528-592dcb82370f\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.464498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.464392    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f1c5f4cf-5b1a-4aad-b528-592dcb82370f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b9496bc86-2xc4x\" (UID: \"f1c5f4cf-5b1a-4aad-b528-592dcb82370f\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.464498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.464478    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f1c5f4cf-5b1a-4aad-b528-592dcb82370f-manifests\") pod \"opendatahub-operator-controller-manager-6b9496bc86-2xc4x\" (UID: \"f1c5f4cf-5b1a-4aad-b528-592dcb82370f\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.464622 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.464527    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1c5f4cf-5b1a-4aad-b528-592dcb82370f-tmp\") pod \"opendatahub-operator-controller-manager-6b9496bc86-2xc4x\" (UID: \"f1c5f4cf-5b1a-4aad-b528-592dcb82370f\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.464622 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.464552    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1c5f4cf-5b1a-4aad-b528-592dcb82370f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b9496bc86-2xc4x\" (UID: \"f1c5f4cf-5b1a-4aad-b528-592dcb82370f\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.474356 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.474330    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"]
Sep 04 23:41:36.565520 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.565489    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25v4m\" (UniqueName: \"kubernetes.io/projected/f1c5f4cf-5b1a-4aad-b528-592dcb82370f-kube-api-access-25v4m\") pod \"opendatahub-operator-controller-manager-6b9496bc86-2xc4x\" (UID: \"f1c5f4cf-5b1a-4aad-b528-592dcb82370f\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.565699 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.565532    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f1c5f4cf-5b1a-4aad-b528-592dcb82370f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b9496bc86-2xc4x\" (UID: \"f1c5f4cf-5b1a-4aad-b528-592dcb82370f\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.565699 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.565564    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f1c5f4cf-5b1a-4aad-b528-592dcb82370f-manifests\") pod \"opendatahub-operator-controller-manager-6b9496bc86-2xc4x\" (UID: \"f1c5f4cf-5b1a-4aad-b528-592dcb82370f\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.565699 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.565588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1c5f4cf-5b1a-4aad-b528-592dcb82370f-tmp\") pod \"opendatahub-operator-controller-manager-6b9496bc86-2xc4x\" (UID: \"f1c5f4cf-5b1a-4aad-b528-592dcb82370f\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.565699 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.565603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1c5f4cf-5b1a-4aad-b528-592dcb82370f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b9496bc86-2xc4x\" (UID: \"f1c5f4cf-5b1a-4aad-b528-592dcb82370f\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.566126 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.566084    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f1c5f4cf-5b1a-4aad-b528-592dcb82370f-manifests\") pod \"opendatahub-operator-controller-manager-6b9496bc86-2xc4x\" (UID: \"f1c5f4cf-5b1a-4aad-b528-592dcb82370f\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.567963 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.567934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1c5f4cf-5b1a-4aad-b528-592dcb82370f-tmp\") pod \"opendatahub-operator-controller-manager-6b9496bc86-2xc4x\" (UID: \"f1c5f4cf-5b1a-4aad-b528-592dcb82370f\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.568060 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.568042    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f1c5f4cf-5b1a-4aad-b528-592dcb82370f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b9496bc86-2xc4x\" (UID: \"f1c5f4cf-5b1a-4aad-b528-592dcb82370f\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.568100 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.568057    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1c5f4cf-5b1a-4aad-b528-592dcb82370f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b9496bc86-2xc4x\" (UID: \"f1c5f4cf-5b1a-4aad-b528-592dcb82370f\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.577800 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.577776    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25v4m\" (UniqueName: \"kubernetes.io/projected/f1c5f4cf-5b1a-4aad-b528-592dcb82370f-kube-api-access-25v4m\") pod \"opendatahub-operator-controller-manager-6b9496bc86-2xc4x\" (UID: \"f1c5f4cf-5b1a-4aad-b528-592dcb82370f\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:36.768049 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:36.767957    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:37.046469 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:37.046386    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"]
Sep 04 23:41:37.049459 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:41:37.049432    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c5f4cf_5b1a_4aad_b528_592dcb82370f.slice/crio-9ea2a675618568bbdb574cf8e4017ac57601a5972cb2670b7944cdd6bd6da4ca WatchSource:0}: Error finding container 9ea2a675618568bbdb574cf8e4017ac57601a5972cb2670b7944cdd6bd6da4ca: Status 404 returned error can't find the container with id 9ea2a675618568bbdb574cf8e4017ac57601a5972cb2670b7944cdd6bd6da4ca
Sep 04 23:41:37.051341 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:37.051323    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 23:41:37.841609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:37.841558    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x" event={"ID":"f1c5f4cf-5b1a-4aad-b528-592dcb82370f","Type":"ContainerStarted","Data":"9ea2a675618568bbdb574cf8e4017ac57601a5972cb2670b7944cdd6bd6da4ca"}
Sep 04 23:41:40.860589 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:40.860549    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x" event={"ID":"f1c5f4cf-5b1a-4aad-b528-592dcb82370f","Type":"ContainerStarted","Data":"a6220c5eedd74fd07e15ef424ddc68f984f8a941077e9b8f3b7c856dca81e134"}
Sep 04 23:41:41.866560 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:41.866527    2562 generic.go:356] "Generic (PLEG): container finished" podID="f1c5f4cf-5b1a-4aad-b528-592dcb82370f" containerID="a6220c5eedd74fd07e15ef424ddc68f984f8a941077e9b8f3b7c856dca81e134" exitCode=0
Sep 04 23:41:41.866974 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:41.866601    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x" event={"ID":"f1c5f4cf-5b1a-4aad-b528-592dcb82370f","Type":"ContainerDied","Data":"a6220c5eedd74fd07e15ef424ddc68f984f8a941077e9b8f3b7c856dca81e134"}
Sep 04 23:41:42.872962 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:42.872929    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x" event={"ID":"f1c5f4cf-5b1a-4aad-b528-592dcb82370f","Type":"ContainerStarted","Data":"a7d8746e1d623311b562a183e8719872cd23f81679f7cfc285c1f592b7e9aeb0"}
Sep 04 23:41:42.873439 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:42.873029    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:42.893808 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:42.893764    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x" podStartSLOduration=2.43229574 podStartE2EDuration="6.893745367s" totalImagesPullingTime="3.409562079s" totalInitContainerRuntime="1.051887548s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:41:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:41:42.892833423 +0000 UTC m=+315.139500515" watchObservedRunningTime="2026-09-04 23:41:42.893745367 +0000 UTC m=+315.140412445"
Sep 04 23:41:53.878741 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:53.878709    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-2xc4x"
Sep 04 23:41:56.398676 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.398621    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh"]
Sep 04 23:41:56.404301 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.404282    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh"
Sep 04 23:41:56.407344 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.407325    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 04 23:41:56.407442 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.407328    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 04 23:41:56.414422 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.414399    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh"]
Sep 04 23:41:56.528617 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.528581    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h62s7\" (UniqueName: \"kubernetes.io/projected/c8007e7f-c503-42f0-a3b5-fdd289603da3-kube-api-access-h62s7\") pod \"kube-auth-proxy-5d989d9b5-46mnh\" (UID: \"c8007e7f-c503-42f0-a3b5-fdd289603da3\") " pod="openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh"
Sep 04 23:41:56.528617 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.528621    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8007e7f-c503-42f0-a3b5-fdd289603da3-tls-certs\") pod \"kube-auth-proxy-5d989d9b5-46mnh\" (UID: \"c8007e7f-c503-42f0-a3b5-fdd289603da3\") " pod="openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh"
Sep 04 23:41:56.528830 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.528715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8007e7f-c503-42f0-a3b5-fdd289603da3-tmp\") pod \"kube-auth-proxy-5d989d9b5-46mnh\" (UID: \"c8007e7f-c503-42f0-a3b5-fdd289603da3\") " pod="openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh"
Sep 04 23:41:56.629920 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.629884    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h62s7\" (UniqueName: \"kubernetes.io/projected/c8007e7f-c503-42f0-a3b5-fdd289603da3-kube-api-access-h62s7\") pod \"kube-auth-proxy-5d989d9b5-46mnh\" (UID: \"c8007e7f-c503-42f0-a3b5-fdd289603da3\") " pod="openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh"
Sep 04 23:41:56.629920 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.629923    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8007e7f-c503-42f0-a3b5-fdd289603da3-tls-certs\") pod \"kube-auth-proxy-5d989d9b5-46mnh\" (UID: \"c8007e7f-c503-42f0-a3b5-fdd289603da3\") " pod="openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh"
Sep 04 23:41:56.630128 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.629965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8007e7f-c503-42f0-a3b5-fdd289603da3-tmp\") pod \"kube-auth-proxy-5d989d9b5-46mnh\" (UID: \"c8007e7f-c503-42f0-a3b5-fdd289603da3\") " pod="openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh"
Sep 04 23:41:56.632258 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.632227    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8007e7f-c503-42f0-a3b5-fdd289603da3-tmp\") pod \"kube-auth-proxy-5d989d9b5-46mnh\" (UID: \"c8007e7f-c503-42f0-a3b5-fdd289603da3\") " pod="openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh"
Sep 04 23:41:56.632458 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.632438    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8007e7f-c503-42f0-a3b5-fdd289603da3-tls-certs\") pod \"kube-auth-proxy-5d989d9b5-46mnh\" (UID: \"c8007e7f-c503-42f0-a3b5-fdd289603da3\") " pod="openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh"
Sep 04 23:41:56.648450 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.648427    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h62s7\" (UniqueName: \"kubernetes.io/projected/c8007e7f-c503-42f0-a3b5-fdd289603da3-kube-api-access-h62s7\") pod \"kube-auth-proxy-5d989d9b5-46mnh\" (UID: \"c8007e7f-c503-42f0-a3b5-fdd289603da3\") " pod="openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh"
Sep 04 23:41:56.713573 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.713537    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh"
Sep 04 23:41:56.995719 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:56.995598    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh"]
Sep 04 23:41:56.997993 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:41:56.997962    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8007e7f_c503_42f0_a3b5_fdd289603da3.slice/crio-ed1e6aca0b28dda99d8fe5f2b418ddca41a12ed37349884bec835264123e6bf4 WatchSource:0}: Error finding container ed1e6aca0b28dda99d8fe5f2b418ddca41a12ed37349884bec835264123e6bf4: Status 404 returned error can't find the container with id ed1e6aca0b28dda99d8fe5f2b418ddca41a12ed37349884bec835264123e6bf4
Sep 04 23:41:57.644824 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.644793    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"]
Sep 04 23:41:57.647499 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.647473    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.649667 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.649631    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 04 23:41:57.649913 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.649893    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 04 23:41:57.650179 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.650156    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 04 23:41:57.650179 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.650174    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-c8td6\""
Sep 04 23:41:57.650339 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.650180    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 04 23:41:57.662815 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.662787    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"]
Sep 04 23:41:57.743576 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.743540    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.743781 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.743595    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.743851 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.743777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks96f\" (UniqueName: \"kubernetes.io/projected/2225c3f0-22d7-4165-9f68-79a9202b1dfb-kube-api-access-ks96f\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.743908 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.743846    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.743960 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.743907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.743960 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.743934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2225c3f0-22d7-4165-9f68-79a9202b1dfb-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.744051 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.743964    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.844832 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.844773    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.845030 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.844892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.845030 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.844967    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks96f\" (UniqueName: \"kubernetes.io/projected/2225c3f0-22d7-4165-9f68-79a9202b1dfb-kube-api-access-ks96f\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.845030 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.845011    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.845189 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.845069    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.845189 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.845098    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2225c3f0-22d7-4165-9f68-79a9202b1dfb-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.845189 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.845130    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.845837 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.845784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.847904 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.847874    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.848045 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.848023    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.848196 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.848173    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2225c3f0-22d7-4165-9f68-79a9202b1dfb-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.849019 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.848993    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.858983 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.858954    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.858983 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.858977    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks96f\" (UniqueName: \"kubernetes.io/projected/2225c3f0-22d7-4165-9f68-79a9202b1dfb-kube-api-access-ks96f\") pod \"istiod-openshift-gateway-7b7995d5cd-89fb4\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:57.937211 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.937124    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh" event={"ID":"c8007e7f-c503-42f0-a3b5-fdd289603da3","Type":"ContainerStarted","Data":"ed1e6aca0b28dda99d8fe5f2b418ddca41a12ed37349884bec835264123e6bf4"}
Sep 04 23:41:57.960735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:57.960699    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:41:58.250770 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:58.250730    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"]
Sep 04 23:41:58.255097 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:41:58.254885    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2225c3f0_22d7_4165_9f68_79a9202b1dfb.slice/crio-ac04b7331b5e8e9d363132634711a8f4b7b057e916c6f305bdad90ba2d602886 WatchSource:0}: Error finding container ac04b7331b5e8e9d363132634711a8f4b7b057e916c6f305bdad90ba2d602886: Status 404 returned error can't find the container with id ac04b7331b5e8e9d363132634711a8f4b7b057e916c6f305bdad90ba2d602886
Sep 04 23:41:58.944763 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:58.944725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4" event={"ID":"2225c3f0-22d7-4165-9f68-79a9202b1dfb","Type":"ContainerStarted","Data":"ac04b7331b5e8e9d363132634711a8f4b7b057e916c6f305bdad90ba2d602886"}
Sep 04 23:41:59.952928 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:59.952888    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh" event={"ID":"c8007e7f-c503-42f0-a3b5-fdd289603da3","Type":"ContainerStarted","Data":"fb35a5707cccf091fea988a3238bce5edc6f10ae901f82601cb24fb140ff8398"}
Sep 04 23:41:59.969881 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:41:59.969821    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d989d9b5-46mnh" podStartSLOduration=1.24222468 podStartE2EDuration="3.969802187s" totalImagesPullingTime="2.727577507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:41:59.968457497 +0000 UTC m=+332.215124577" watchObservedRunningTime="2026-09-04 23:41:59.969802187 +0000 UTC m=+332.216469265"
Sep 04 23:42:01.507249 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:01.507211    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:42:01.507506 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:01.507286    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:42:01.964250 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:01.964207    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4" event={"ID":"2225c3f0-22d7-4165-9f68-79a9202b1dfb","Type":"ContainerStarted","Data":"7729b315c153c362879bd9a849c784024d1fd829c21cbe8a8f4f2de50330f895"}
Sep 04 23:42:01.964422 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:01.964301    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:42:01.982303 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:01.982247    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4" podStartSLOduration=1.733242707 podStartE2EDuration="4.982233749s" totalImagesPullingTime="3.248991042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:42:01.981026409 +0000 UTC m=+334.227693488" watchObservedRunningTime="2026-09-04 23:42:01.982233749 +0000 UTC m=+334.228900828"
Sep 04 23:42:02.971924 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:02.971892    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:42:17.011643 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:17.011604    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-57t66"]
Sep 04 23:42:17.016742 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:17.016720    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-57t66"
Sep 04 23:42:17.019410 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:17.019386    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 04 23:42:17.019526 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:17.019473    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-tlttq\""
Sep 04 23:42:17.027965 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:17.027359    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-57t66"]
Sep 04 23:42:17.114173 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:17.114125    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f180203c-534c-4b4a-bc01-fb6885e26f22-cert\") pod \"kserve-controller-manager-d6f4b65b8-57t66\" (UID: \"f180203c-534c-4b4a-bc01-fb6885e26f22\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-57t66"
Sep 04 23:42:17.114369 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:17.114200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4vx9\" (UniqueName: \"kubernetes.io/projected/f180203c-534c-4b4a-bc01-fb6885e26f22-kube-api-access-r4vx9\") pod \"kserve-controller-manager-d6f4b65b8-57t66\" (UID: \"f180203c-534c-4b4a-bc01-fb6885e26f22\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-57t66"
Sep 04 23:42:17.215582 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:17.215537    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f180203c-534c-4b4a-bc01-fb6885e26f22-cert\") pod \"kserve-controller-manager-d6f4b65b8-57t66\" (UID: \"f180203c-534c-4b4a-bc01-fb6885e26f22\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-57t66"
Sep 04 23:42:17.215784 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:17.215593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4vx9\" (UniqueName: \"kubernetes.io/projected/f180203c-534c-4b4a-bc01-fb6885e26f22-kube-api-access-r4vx9\") pod \"kserve-controller-manager-d6f4b65b8-57t66\" (UID: \"f180203c-534c-4b4a-bc01-fb6885e26f22\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-57t66"
Sep 04 23:42:17.218051 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:17.218019    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f180203c-534c-4b4a-bc01-fb6885e26f22-cert\") pod \"kserve-controller-manager-d6f4b65b8-57t66\" (UID: \"f180203c-534c-4b4a-bc01-fb6885e26f22\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-57t66"
Sep 04 23:42:17.223755 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:17.223731    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4vx9\" (UniqueName: \"kubernetes.io/projected/f180203c-534c-4b4a-bc01-fb6885e26f22-kube-api-access-r4vx9\") pod \"kserve-controller-manager-d6f4b65b8-57t66\" (UID: \"f180203c-534c-4b4a-bc01-fb6885e26f22\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-57t66"
Sep 04 23:42:17.330076 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:17.329996    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-57t66"
Sep 04 23:42:17.598347 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:17.598323    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-57t66"]
Sep 04 23:42:17.600452 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:42:17.600422    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf180203c_534c_4b4a_bc01_fb6885e26f22.slice/crio-ba9f7c5ae8f43660f8b26cf9d6dc791e78c3ee39d54ccc33c9537033379719c9 WatchSource:0}: Error finding container ba9f7c5ae8f43660f8b26cf9d6dc791e78c3ee39d54ccc33c9537033379719c9: Status 404 returned error can't find the container with id ba9f7c5ae8f43660f8b26cf9d6dc791e78c3ee39d54ccc33c9537033379719c9
Sep 04 23:42:18.040249 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:18.040216    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-57t66" event={"ID":"f180203c-534c-4b4a-bc01-fb6885e26f22","Type":"ContainerStarted","Data":"ba9f7c5ae8f43660f8b26cf9d6dc791e78c3ee39d54ccc33c9537033379719c9"}
Sep 04 23:42:20.050072 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:20.050037    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-57t66" event={"ID":"f180203c-534c-4b4a-bc01-fb6885e26f22","Type":"ContainerStarted","Data":"55419bb50e75b0cd574c81280d18739f07b1e957d1ada1ed0e3964dce13bc375"}
Sep 04 23:42:20.050480 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:20.050199    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-57t66"
Sep 04 23:42:20.066708 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:20.066629    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-57t66" podStartSLOduration=2.078916378 podStartE2EDuration="4.066613658s" totalImagesPullingTime="1.98769728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:42:20.065403686 +0000 UTC m=+352.312070772" watchObservedRunningTime="2026-09-04 23:42:20.066613658 +0000 UTC m=+352.313280738"
Sep 04 23:42:22.545685 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.545600    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"]
Sep 04 23:42:22.546077 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.545796    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4" podUID="2225c3f0-22d7-4165-9f68-79a9202b1dfb" containerName="discovery" containerID="cri-o://7729b315c153c362879bd9a849c784024d1fd829c21cbe8a8f4f2de50330f895" gracePeriod=30
Sep 04 23:42:22.803318 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.803263    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:42:22.965733 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.965694    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-token\") pod \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") "
Sep 04 23:42:22.965922 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.965748    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-kubeconfig\") pod \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") "
Sep 04 23:42:22.965922 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.965787    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-csr-ca-configmap\") pod \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") "
Sep 04 23:42:22.965922 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.965840    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks96f\" (UniqueName: \"kubernetes.io/projected/2225c3f0-22d7-4165-9f68-79a9202b1dfb-kube-api-access-ks96f\") pod \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") "
Sep 04 23:42:22.965922 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.965876    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-cacerts\") pod \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") "
Sep 04 23:42:22.965922 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.965911    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2225c3f0-22d7-4165-9f68-79a9202b1dfb-local-certs\") pod \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") "
Sep 04 23:42:22.966166 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.965956    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-csr-dns-cert\") pod \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\" (UID: \"2225c3f0-22d7-4165-9f68-79a9202b1dfb\") "
Sep 04 23:42:22.966271 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.966229    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "2225c3f0-22d7-4165-9f68-79a9202b1dfb" (UID: "2225c3f0-22d7-4165-9f68-79a9202b1dfb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:42:22.968351 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.968316    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2225c3f0-22d7-4165-9f68-79a9202b1dfb-kube-api-access-ks96f" (OuterVolumeSpecName: "kube-api-access-ks96f") pod "2225c3f0-22d7-4165-9f68-79a9202b1dfb" (UID: "2225c3f0-22d7-4165-9f68-79a9202b1dfb"). InnerVolumeSpecName "kube-api-access-ks96f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:42:22.968523 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.968367    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-token" (OuterVolumeSpecName: "istio-token") pod "2225c3f0-22d7-4165-9f68-79a9202b1dfb" (UID: "2225c3f0-22d7-4165-9f68-79a9202b1dfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:42:22.968523 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.968379    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "2225c3f0-22d7-4165-9f68-79a9202b1dfb" (UID: "2225c3f0-22d7-4165-9f68-79a9202b1dfb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:42:22.968626 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.968587    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2225c3f0-22d7-4165-9f68-79a9202b1dfb-local-certs" (OuterVolumeSpecName: "local-certs") pod "2225c3f0-22d7-4165-9f68-79a9202b1dfb" (UID: "2225c3f0-22d7-4165-9f68-79a9202b1dfb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:42:22.968702 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.968626    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "2225c3f0-22d7-4165-9f68-79a9202b1dfb" (UID: "2225c3f0-22d7-4165-9f68-79a9202b1dfb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:42:22.968741 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:22.968698    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-cacerts" (OuterVolumeSpecName: "cacerts") pod "2225c3f0-22d7-4165-9f68-79a9202b1dfb" (UID: "2225c3f0-22d7-4165-9f68-79a9202b1dfb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:42:23.064904 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.064812    2562 generic.go:356] "Generic (PLEG): container finished" podID="2225c3f0-22d7-4165-9f68-79a9202b1dfb" containerID="7729b315c153c362879bd9a849c784024d1fd829c21cbe8a8f4f2de50330f895" exitCode=0
Sep 04 23:42:23.064904 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.064857    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"
Sep 04 23:42:23.065089 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.064905    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4" event={"ID":"2225c3f0-22d7-4165-9f68-79a9202b1dfb","Type":"ContainerDied","Data":"7729b315c153c362879bd9a849c784024d1fd829c21cbe8a8f4f2de50330f895"}
Sep 04 23:42:23.065089 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.064954    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4" event={"ID":"2225c3f0-22d7-4165-9f68-79a9202b1dfb","Type":"ContainerDied","Data":"ac04b7331b5e8e9d363132634711a8f4b7b057e916c6f305bdad90ba2d602886"}
Sep 04 23:42:23.065089 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.064975    2562 scope.go:117] "RemoveContainer" containerID="7729b315c153c362879bd9a849c784024d1fd829c21cbe8a8f4f2de50330f895"
Sep 04 23:42:23.067115 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.067091    2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-kubeconfig\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:42:23.067115 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.067118    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-csr-ca-configmap\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:42:23.067304 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.067132    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks96f\" (UniqueName: \"kubernetes.io/projected/2225c3f0-22d7-4165-9f68-79a9202b1dfb-kube-api-access-ks96f\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:42:23.067304 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.067147    2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-cacerts\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:42:23.067304 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.067160    2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2225c3f0-22d7-4165-9f68-79a9202b1dfb-local-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:42:23.067304 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.067173    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-csr-dns-cert\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:42:23.067304 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.067186    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2225c3f0-22d7-4165-9f68-79a9202b1dfb-istio-token\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:42:23.074107 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.074093    2562 scope.go:117] "RemoveContainer" containerID="7729b315c153c362879bd9a849c784024d1fd829c21cbe8a8f4f2de50330f895"
Sep 04 23:42:23.074396 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:42:23.074379    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7729b315c153c362879bd9a849c784024d1fd829c21cbe8a8f4f2de50330f895\": container with ID starting with 7729b315c153c362879bd9a849c784024d1fd829c21cbe8a8f4f2de50330f895 not found: ID does not exist" containerID="7729b315c153c362879bd9a849c784024d1fd829c21cbe8a8f4f2de50330f895"
Sep 04 23:42:23.074434 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.074404    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7729b315c153c362879bd9a849c784024d1fd829c21cbe8a8f4f2de50330f895"} err="failed to get container status \"7729b315c153c362879bd9a849c784024d1fd829c21cbe8a8f4f2de50330f895\": rpc error: code = NotFound desc = could not find container \"7729b315c153c362879bd9a849c784024d1fd829c21cbe8a8f4f2de50330f895\": container with ID starting with 7729b315c153c362879bd9a849c784024d1fd829c21cbe8a8f4f2de50330f895 not found: ID does not exist"
Sep 04 23:42:23.091249 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.091221    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"]
Sep 04 23:42:23.092097 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:23.092075    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-89fb4"]
Sep 04 23:42:24.359782 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:24.359748    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2225c3f0-22d7-4165-9f68-79a9202b1dfb" path="/var/lib/kubelet/pods/2225c3f0-22d7-4165-9f68-79a9202b1dfb/volumes"
Sep 04 23:42:51.060281 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:42:51.060247    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-57t66"
Sep 04 23:43:24.264187 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.264154    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"]
Sep 04 23:43:24.264645 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.264489    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2225c3f0-22d7-4165-9f68-79a9202b1dfb" containerName="discovery"
Sep 04 23:43:24.264645 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.264499    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2225c3f0-22d7-4165-9f68-79a9202b1dfb" containerName="discovery"
Sep 04 23:43:24.264645 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.264553    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2225c3f0-22d7-4165-9f68-79a9202b1dfb" containerName="discovery"
Sep 04 23:43:24.267388 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.267373    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:24.269715 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.269687    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 04 23:43:24.269841 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.269712    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 04 23:43:24.269841 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.269776    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j28nh\""
Sep 04 23:43:24.280244 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.280222    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"]
Sep 04 23:43:24.372740 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.372705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4903539c-9c87-4c00-bca4-91a544602427-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" (UID: \"4903539c-9c87-4c00-bca4-91a544602427\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:24.372740 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.372740    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjnc\" (UniqueName: \"kubernetes.io/projected/4903539c-9c87-4c00-bca4-91a544602427-kube-api-access-xmjnc\") pod \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" (UID: \"4903539c-9c87-4c00-bca4-91a544602427\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:24.473184 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.473146    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4903539c-9c87-4c00-bca4-91a544602427-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" (UID: \"4903539c-9c87-4c00-bca4-91a544602427\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:24.473184 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.473182    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjnc\" (UniqueName: \"kubernetes.io/projected/4903539c-9c87-4c00-bca4-91a544602427-kube-api-access-xmjnc\") pod \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" (UID: \"4903539c-9c87-4c00-bca4-91a544602427\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:24.473515 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.473497    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4903539c-9c87-4c00-bca4-91a544602427-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" (UID: \"4903539c-9c87-4c00-bca4-91a544602427\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:24.481620 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.481591    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjnc\" (UniqueName: \"kubernetes.io/projected/4903539c-9c87-4c00-bca4-91a544602427-kube-api-access-xmjnc\") pod \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" (UID: \"4903539c-9c87-4c00-bca4-91a544602427\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:24.577723 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.577625    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:24.856028 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:24.855923    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"]
Sep 04 23:43:24.859221 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:43:24.859190    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4903539c_9c87_4c00_bca4_91a544602427.slice/crio-a863b991358d7698988583b61ac25e790da7bd5256cab1825c8c2ccef2680497 WatchSource:0}: Error finding container a863b991358d7698988583b61ac25e790da7bd5256cab1825c8c2ccef2680497: Status 404 returned error can't find the container with id a863b991358d7698988583b61ac25e790da7bd5256cab1825c8c2ccef2680497
Sep 04 23:43:25.328860 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:25.328823    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s" event={"ID":"4903539c-9c87-4c00-bca4-91a544602427","Type":"ContainerStarted","Data":"a863b991358d7698988583b61ac25e790da7bd5256cab1825c8c2ccef2680497"}
Sep 04 23:43:26.761544 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:26.761501    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-zggq7"]
Sep 04 23:43:26.770583 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:26.770555    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-zggq7"
Sep 04 23:43:26.773178 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:26.773155    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-lbl7x\""
Sep 04 23:43:26.779902 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:26.779877    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-zggq7"]
Sep 04 23:43:26.791127 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:26.791100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rcrf\" (UniqueName: \"kubernetes.io/projected/74e16c05-c1ed-4af2-92f7-d7c85f6d8ce2-kube-api-access-8rcrf\") pod \"authorino-operator-68d88948d8-zggq7\" (UID: \"74e16c05-c1ed-4af2-92f7-d7c85f6d8ce2\") " pod="kuadrant-system/authorino-operator-68d88948d8-zggq7"
Sep 04 23:43:26.891560 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:26.891517    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rcrf\" (UniqueName: \"kubernetes.io/projected/74e16c05-c1ed-4af2-92f7-d7c85f6d8ce2-kube-api-access-8rcrf\") pod \"authorino-operator-68d88948d8-zggq7\" (UID: \"74e16c05-c1ed-4af2-92f7-d7c85f6d8ce2\") " pod="kuadrant-system/authorino-operator-68d88948d8-zggq7"
Sep 04 23:43:26.901586 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:26.901550    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rcrf\" (UniqueName: \"kubernetes.io/projected/74e16c05-c1ed-4af2-92f7-d7c85f6d8ce2-kube-api-access-8rcrf\") pod \"authorino-operator-68d88948d8-zggq7\" (UID: \"74e16c05-c1ed-4af2-92f7-d7c85f6d8ce2\") " pod="kuadrant-system/authorino-operator-68d88948d8-zggq7"
Sep 04 23:43:27.085275 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:27.085184    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-zggq7"
Sep 04 23:43:27.379933 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:27.379901    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-zggq7"]
Sep 04 23:43:27.382873 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:43:27.382839    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e16c05_c1ed_4af2_92f7_d7c85f6d8ce2.slice/crio-6e839c0723ec05b261c4f9b51acfb946ff80035fec465d575f5840ad19da0c95 WatchSource:0}: Error finding container 6e839c0723ec05b261c4f9b51acfb946ff80035fec465d575f5840ad19da0c95: Status 404 returned error can't find the container with id 6e839c0723ec05b261c4f9b51acfb946ff80035fec465d575f5840ad19da0c95
Sep 04 23:43:28.349214 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:28.349180    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-zggq7" event={"ID":"74e16c05-c1ed-4af2-92f7-d7c85f6d8ce2","Type":"ContainerStarted","Data":"6e839c0723ec05b261c4f9b51acfb946ff80035fec465d575f5840ad19da0c95"}
Sep 04 23:43:28.938899 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:28.938867    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"]
Sep 04 23:43:28.942448 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:28.942429    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:28.944940 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:28.944921    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-spddv\""
Sep 04 23:43:28.954957 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:28.954936    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"]
Sep 04 23:43:29.007714 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:29.007677    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcfhs\" (UniqueName: \"kubernetes.io/projected/42ebc494-ce47-46e5-86f8-eccdcab94f58-kube-api-access-qcfhs\") pod \"limitador-operator-controller-manager-6c549b6d85-zm7pm\" (UID: \"42ebc494-ce47-46e5-86f8-eccdcab94f58\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:29.109213 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:29.109115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcfhs\" (UniqueName: \"kubernetes.io/projected/42ebc494-ce47-46e5-86f8-eccdcab94f58-kube-api-access-qcfhs\") pod \"limitador-operator-controller-manager-6c549b6d85-zm7pm\" (UID: \"42ebc494-ce47-46e5-86f8-eccdcab94f58\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:29.119271 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:29.119243    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcfhs\" (UniqueName: \"kubernetes.io/projected/42ebc494-ce47-46e5-86f8-eccdcab94f58-kube-api-access-qcfhs\") pod \"limitador-operator-controller-manager-6c549b6d85-zm7pm\" (UID: \"42ebc494-ce47-46e5-86f8-eccdcab94f58\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:29.254265 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:29.254226    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:29.357552 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:29.357517    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s" event={"ID":"4903539c-9c87-4c00-bca4-91a544602427","Type":"ContainerStarted","Data":"2f5f63747b318babdc96af2a168d23e94d19eba743ecbfe2b56299a118a359fe"}
Sep 04 23:43:29.358019 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:29.357763    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:29.377308 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:29.377206    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s" podStartSLOduration=1.461863712 podStartE2EDuration="5.377183829s" totalImagesPullingTime="3.915320117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:43:29.376419034 +0000 UTC m=+421.623086127" watchObservedRunningTime="2026-09-04 23:43:29.377183829 +0000 UTC m=+421.623850910"
Sep 04 23:43:29.733823 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:29.733793    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"]
Sep 04 23:43:29.735663 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:43:29.735624    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ebc494_ce47_46e5_86f8_eccdcab94f58.slice/crio-d1271386ff0c9a2b82f878f011af4be280d5131b3b881ac9d91e06300cb09509 WatchSource:0}: Error finding container d1271386ff0c9a2b82f878f011af4be280d5131b3b881ac9d91e06300cb09509: Status 404 returned error can't find the container with id d1271386ff0c9a2b82f878f011af4be280d5131b3b881ac9d91e06300cb09509
Sep 04 23:43:30.362917 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:30.362882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-zggq7" event={"ID":"74e16c05-c1ed-4af2-92f7-d7c85f6d8ce2","Type":"ContainerStarted","Data":"49f1c2337c680a2413a704e78b99a640af7e6ba2899a7708a9604553fb946e28"}
Sep 04 23:43:30.363404 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:30.363015    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-zggq7"
Sep 04 23:43:30.364478 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:30.364453    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm" event={"ID":"42ebc494-ce47-46e5-86f8-eccdcab94f58","Type":"ContainerStarted","Data":"d1271386ff0c9a2b82f878f011af4be280d5131b3b881ac9d91e06300cb09509"}
Sep 04 23:43:30.381184 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:30.381141    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-zggq7" podStartSLOduration=2.234590195 podStartE2EDuration="4.381130057s" totalImagesPullingTime="2.146539862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:43:30.37951237 +0000 UTC m=+422.626179450" watchObservedRunningTime="2026-09-04 23:43:30.381130057 +0000 UTC m=+422.627797135"
Sep 04 23:43:31.167077 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.167041    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-gl4w2"]
Sep 04 23:43:31.170350 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.170334    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-gl4w2"
Sep 04 23:43:31.172746 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.172724    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 04 23:43:31.172866 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.172761    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-b6mgp\""
Sep 04 23:43:31.183261 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.183239    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-gl4w2"]
Sep 04 23:43:31.227298 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.227260    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8tln\" (UniqueName: \"kubernetes.io/projected/cf265509-3e29-4c30-aee7-28df1e5912b9-kube-api-access-w8tln\") pod \"dns-operator-controller-manager-5985564776-gl4w2\" (UID: \"cf265509-3e29-4c30-aee7-28df1e5912b9\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-gl4w2"
Sep 04 23:43:31.328479 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.328437    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8tln\" (UniqueName: \"kubernetes.io/projected/cf265509-3e29-4c30-aee7-28df1e5912b9-kube-api-access-w8tln\") pod \"dns-operator-controller-manager-5985564776-gl4w2\" (UID: \"cf265509-3e29-4c30-aee7-28df1e5912b9\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-gl4w2"
Sep 04 23:43:31.339305 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.339273    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8tln\" (UniqueName: \"kubernetes.io/projected/cf265509-3e29-4c30-aee7-28df1e5912b9-kube-api-access-w8tln\") pod \"dns-operator-controller-manager-5985564776-gl4w2\" (UID: \"cf265509-3e29-4c30-aee7-28df1e5912b9\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-gl4w2"
Sep 04 23:43:31.482764 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.482731    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-gl4w2"
Sep 04 23:43:31.871341 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.871302    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c"]
Sep 04 23:43:31.875088 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.875067    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c"
Sep 04 23:43:31.877454 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.877425    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 04 23:43:31.877577 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.877524    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 04 23:43:31.877641 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.877585    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jgrf6\""
Sep 04 23:43:31.883536 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.883514    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c"]
Sep 04 23:43:31.933641 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.933604    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/474c9c83-b3b1-43f2-9863-3b8be53cdd11-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-rtj8c\" (UID: \"474c9c83-b3b1-43f2-9863-3b8be53cdd11\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c"
Sep 04 23:43:31.933824 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.933663    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99dcr\" (UniqueName: \"kubernetes.io/projected/474c9c83-b3b1-43f2-9863-3b8be53cdd11-kube-api-access-99dcr\") pod \"kuadrant-console-plugin-598f86468b-rtj8c\" (UID: \"474c9c83-b3b1-43f2-9863-3b8be53cdd11\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c"
Sep 04 23:43:31.933824 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.933706    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/474c9c83-b3b1-43f2-9863-3b8be53cdd11-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-rtj8c\" (UID: \"474c9c83-b3b1-43f2-9863-3b8be53cdd11\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c"
Sep 04 23:43:31.974669 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:31.974620    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-gl4w2"]
Sep 04 23:43:31.976738 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:43:31.976704    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf265509_3e29_4c30_aee7_28df1e5912b9.slice/crio-d7d5cd86e9b9c5042aef19aeb15a52a3bc0a62819cd7f893648b05f980193b6c WatchSource:0}: Error finding container d7d5cd86e9b9c5042aef19aeb15a52a3bc0a62819cd7f893648b05f980193b6c: Status 404 returned error can't find the container with id d7d5cd86e9b9c5042aef19aeb15a52a3bc0a62819cd7f893648b05f980193b6c
Sep 04 23:43:32.034162 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:32.034066    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/474c9c83-b3b1-43f2-9863-3b8be53cdd11-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-rtj8c\" (UID: \"474c9c83-b3b1-43f2-9863-3b8be53cdd11\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c"
Sep 04 23:43:32.034162 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:32.034111    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99dcr\" (UniqueName: \"kubernetes.io/projected/474c9c83-b3b1-43f2-9863-3b8be53cdd11-kube-api-access-99dcr\") pod \"kuadrant-console-plugin-598f86468b-rtj8c\" (UID: \"474c9c83-b3b1-43f2-9863-3b8be53cdd11\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c"
Sep 04 23:43:32.034162 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:32.034146    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/474c9c83-b3b1-43f2-9863-3b8be53cdd11-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-rtj8c\" (UID: \"474c9c83-b3b1-43f2-9863-3b8be53cdd11\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c"
Sep 04 23:43:32.034379 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:32.034270    2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 04 23:43:32.034379 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:32.034332    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/474c9c83-b3b1-43f2-9863-3b8be53cdd11-plugin-serving-cert podName:474c9c83-b3b1-43f2-9863-3b8be53cdd11 nodeName:}" failed. No retries permitted until 2026-09-04 23:43:32.534310263 +0000 UTC m=+424.780977320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/474c9c83-b3b1-43f2-9863-3b8be53cdd11-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-rtj8c" (UID: "474c9c83-b3b1-43f2-9863-3b8be53cdd11") : secret "plugin-serving-cert" not found
Sep 04 23:43:32.035046 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:32.035023    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/474c9c83-b3b1-43f2-9863-3b8be53cdd11-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-rtj8c\" (UID: \"474c9c83-b3b1-43f2-9863-3b8be53cdd11\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c"
Sep 04 23:43:32.042852 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:32.042827    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99dcr\" (UniqueName: \"kubernetes.io/projected/474c9c83-b3b1-43f2-9863-3b8be53cdd11-kube-api-access-99dcr\") pod \"kuadrant-console-plugin-598f86468b-rtj8c\" (UID: \"474c9c83-b3b1-43f2-9863-3b8be53cdd11\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c"
Sep 04 23:43:32.376525 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:32.376437    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-gl4w2" event={"ID":"cf265509-3e29-4c30-aee7-28df1e5912b9","Type":"ContainerStarted","Data":"d7d5cd86e9b9c5042aef19aeb15a52a3bc0a62819cd7f893648b05f980193b6c"}
Sep 04 23:43:32.377930 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:32.377905    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm" event={"ID":"42ebc494-ce47-46e5-86f8-eccdcab94f58","Type":"ContainerStarted","Data":"defdc5a7958c1fbae27be55b4133cc0dde6c1a3f5091bef95d63b6b6489a1d94"}
Sep 04 23:43:32.378068 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:32.378049    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:32.398182 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:32.398133    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm" podStartSLOduration=2.363583029 podStartE2EDuration="4.398116649s" totalImagesPullingTime="2.03453362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:43:32.397318368 +0000 UTC m=+424.643985460" watchObservedRunningTime="2026-09-04 23:43:32.398116649 +0000 UTC m=+424.644783729"
Sep 04 23:43:32.538155 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:32.538113    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/474c9c83-b3b1-43f2-9863-3b8be53cdd11-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-rtj8c\" (UID: \"474c9c83-b3b1-43f2-9863-3b8be53cdd11\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c"
Sep 04 23:43:32.540497 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:32.540475    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/474c9c83-b3b1-43f2-9863-3b8be53cdd11-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-rtj8c\" (UID: \"474c9c83-b3b1-43f2-9863-3b8be53cdd11\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c"
Sep 04 23:43:32.785304 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:32.785268    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c"
Sep 04 23:43:33.066743 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:33.066713    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c"]
Sep 04 23:43:33.068320 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:43:33.068292    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod474c9c83_b3b1_43f2_9863_3b8be53cdd11.slice/crio-e4ed0787a7a91c8100932dc7272ab55d6a60094693aa6b35f3a62a51c7c0e6f0 WatchSource:0}: Error finding container e4ed0787a7a91c8100932dc7272ab55d6a60094693aa6b35f3a62a51c7c0e6f0: Status 404 returned error can't find the container with id e4ed0787a7a91c8100932dc7272ab55d6a60094693aa6b35f3a62a51c7c0e6f0
Sep 04 23:43:33.384705 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:33.384605    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c" event={"ID":"474c9c83-b3b1-43f2-9863-3b8be53cdd11","Type":"ContainerStarted","Data":"e4ed0787a7a91c8100932dc7272ab55d6a60094693aa6b35f3a62a51c7c0e6f0"}
Sep 04 23:43:34.393167 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:34.393126    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-gl4w2" event={"ID":"cf265509-3e29-4c30-aee7-28df1e5912b9","Type":"ContainerStarted","Data":"ae089f50ab6022415c08bac9d7080c83ad0282581a332fc5714bdce465e74e86"}
Sep 04 23:43:34.393637 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:34.393217    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-gl4w2"
Sep 04 23:43:34.412343 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:34.412288    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-gl4w2" podStartSLOduration=1.726859729 podStartE2EDuration="3.412267414s" totalImagesPullingTime="1.685407685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:43:34.410359042 +0000 UTC m=+426.657026122" watchObservedRunningTime="2026-09-04 23:43:34.412267414 +0000 UTC m=+426.658934494"
Sep 04 23:43:38.416397 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:38.416366    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c" event={"ID":"474c9c83-b3b1-43f2-9863-3b8be53cdd11","Type":"ContainerStarted","Data":"6ca1f0b6af621e356adef20158acce19d655c76d02f6f26cc666039868bb5b59"}
Sep 04 23:43:38.432063 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:38.432019    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rtj8c" podStartSLOduration=3.048558802 podStartE2EDuration="7.432004022s" totalImagesPullingTime="4.38344522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:43:38.430351798 +0000 UTC m=+430.677018878" watchObservedRunningTime="2026-09-04 23:43:38.432004022 +0000 UTC m=+430.678671100"
Sep 04 23:43:40.366600 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:40.366570    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:41.373311 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:41.373282    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-zggq7"
Sep 04 23:43:42.360666 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.360625    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"]
Sep 04 23:43:42.360841 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.360802    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s" podUID="4903539c-9c87-4c00-bca4-91a544602427" containerName="manager" containerID="cri-o://2f5f63747b318babdc96af2a168d23e94d19eba743ecbfe2b56299a118a359fe" gracePeriod=2
Sep 04 23:43:42.362875 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:42.362842    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="4903539c-9c87-4c00-bca4-91a544602427" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:42.364193 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.364161    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"]
Sep 04 23:43:42.377294 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.377267    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"]
Sep 04 23:43:42.377716 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.377476    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm" podUID="42ebc494-ce47-46e5-86f8-eccdcab94f58" containerName="manager" containerID="cri-o://defdc5a7958c1fbae27be55b4133cc0dde6c1a3f5091bef95d63b6b6489a1d94" gracePeriod=2
Sep 04 23:43:42.379733 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.379707    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:42.387638 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.387613    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"]
Sep 04 23:43:42.388213 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.388190    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4903539c-9c87-4c00-bca4-91a544602427" containerName="manager"
Sep 04 23:43:42.388325 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.388217    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4903539c-9c87-4c00-bca4-91a544602427" containerName="manager"
Sep 04 23:43:42.388325 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.388313    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4903539c-9c87-4c00-bca4-91a544602427" containerName="manager"
Sep 04 23:43:42.388765 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:42.388731    2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"42ebc494-ce47-46e5-86f8-eccdcab94f58\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-6c549b6d85-zm7pm\": pods \"limitador-operator-controller-manager-6c549b6d85-zm7pm\" not found" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:42.390820 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:42.390794    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="4903539c-9c87-4c00-bca4-91a544602427" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:42.392557 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:42.392529    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="4903539c-9c87-4c00-bca4-91a544602427" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:42.394144 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:42.394113    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zm7pm\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="42ebc494-ce47-46e5-86f8-eccdcab94f58" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:42.396035 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.396016    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"]
Sep 04 23:43:42.396132 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.396121    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"
Sep 04 23:43:42.397585 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.397561    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-m4dvc"]
Sep 04 23:43:42.398126 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.398056    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ebc494-ce47-46e5-86f8-eccdcab94f58" containerName="manager"
Sep 04 23:43:42.398126 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.398087    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ebc494-ce47-46e5-86f8-eccdcab94f58" containerName="manager"
Sep 04 23:43:42.398325 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:42.398167    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zm7pm\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="42ebc494-ce47-46e5-86f8-eccdcab94f58" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:42.398325 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.398194    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="42ebc494-ce47-46e5-86f8-eccdcab94f58" containerName="manager"
Sep 04 23:43:42.402505 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.402481    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-m4dvc"
Sep 04 23:43:42.404804 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.404779    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"]
Sep 04 23:43:42.414287 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.414263    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-m4dvc"]
Sep 04 23:43:42.422475 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.422448    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37056ead-63b9-498f-8622-feb8fd337188-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-hnpqk\" (UID: \"37056ead-63b9-498f-8622-feb8fd337188\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"
Sep 04 23:43:42.422603 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.422491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r29qz\" (UniqueName: \"kubernetes.io/projected/632a7bdc-5cb2-453c-87fb-72557a347170-kube-api-access-r29qz\") pod \"limitador-operator-controller-manager-6c549b6d85-m4dvc\" (UID: \"632a7bdc-5cb2-453c-87fb-72557a347170\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-m4dvc"
Sep 04 23:43:42.422603 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.422525    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vzph\" (UniqueName: \"kubernetes.io/projected/37056ead-63b9-498f-8622-feb8fd337188-kube-api-access-8vzph\") pod \"kuadrant-operator-controller-manager-58fc56757c-hnpqk\" (UID: \"37056ead-63b9-498f-8622-feb8fd337188\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"
Sep 04 23:43:42.423412 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:42.423388    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="4903539c-9c87-4c00-bca4-91a544602427" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:42.425195 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:42.425172    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zm7pm\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="42ebc494-ce47-46e5-86f8-eccdcab94f58" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:42.445770 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:42.445744    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="4903539c-9c87-4c00-bca4-91a544602427" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:42.523418 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.523337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37056ead-63b9-498f-8622-feb8fd337188-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-hnpqk\" (UID: \"37056ead-63b9-498f-8622-feb8fd337188\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"
Sep 04 23:43:42.523418 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.523378    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r29qz\" (UniqueName: \"kubernetes.io/projected/632a7bdc-5cb2-453c-87fb-72557a347170-kube-api-access-r29qz\") pod \"limitador-operator-controller-manager-6c549b6d85-m4dvc\" (UID: \"632a7bdc-5cb2-453c-87fb-72557a347170\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-m4dvc"
Sep 04 23:43:42.523418 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.523409    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vzph\" (UniqueName: \"kubernetes.io/projected/37056ead-63b9-498f-8622-feb8fd337188-kube-api-access-8vzph\") pod \"kuadrant-operator-controller-manager-58fc56757c-hnpqk\" (UID: \"37056ead-63b9-498f-8622-feb8fd337188\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"
Sep 04 23:43:42.523902 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.523879    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37056ead-63b9-498f-8622-feb8fd337188-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-hnpqk\" (UID: \"37056ead-63b9-498f-8622-feb8fd337188\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"
Sep 04 23:43:42.538229 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.538172    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r29qz\" (UniqueName: \"kubernetes.io/projected/632a7bdc-5cb2-453c-87fb-72557a347170-kube-api-access-r29qz\") pod \"limitador-operator-controller-manager-6c549b6d85-m4dvc\" (UID: \"632a7bdc-5cb2-453c-87fb-72557a347170\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-m4dvc"
Sep 04 23:43:42.539162 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.539114    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vzph\" (UniqueName: \"kubernetes.io/projected/37056ead-63b9-498f-8622-feb8fd337188-kube-api-access-8vzph\") pod \"kuadrant-operator-controller-manager-58fc56757c-hnpqk\" (UID: \"37056ead-63b9-498f-8622-feb8fd337188\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"
Sep 04 23:43:42.610272 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.610247    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:42.612681 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:42.612598    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zm7pm\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="42ebc494-ce47-46e5-86f8-eccdcab94f58" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:42.614171 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.614153    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:42.614558 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:42.614537    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="4903539c-9c87-4c00-bca4-91a544602427" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:42.616412 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:42.616391    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zm7pm\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="42ebc494-ce47-46e5-86f8-eccdcab94f58" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:42.618011 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:42.617990    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="4903539c-9c87-4c00-bca4-91a544602427" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:42.724268 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.724231    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4903539c-9c87-4c00-bca4-91a544602427-extensions-socket-volume\") pod \"4903539c-9c87-4c00-bca4-91a544602427\" (UID: \"4903539c-9c87-4c00-bca4-91a544602427\") "
Sep 04 23:43:42.724460 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.724282    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmjnc\" (UniqueName: \"kubernetes.io/projected/4903539c-9c87-4c00-bca4-91a544602427-kube-api-access-xmjnc\") pod \"4903539c-9c87-4c00-bca4-91a544602427\" (UID: \"4903539c-9c87-4c00-bca4-91a544602427\") "
Sep 04 23:43:42.724460 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.724390    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcfhs\" (UniqueName: \"kubernetes.io/projected/42ebc494-ce47-46e5-86f8-eccdcab94f58-kube-api-access-qcfhs\") pod \"42ebc494-ce47-46e5-86f8-eccdcab94f58\" (UID: \"42ebc494-ce47-46e5-86f8-eccdcab94f58\") "
Sep 04 23:43:42.724814 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.724784    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4903539c-9c87-4c00-bca4-91a544602427-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4903539c-9c87-4c00-bca4-91a544602427" (UID: "4903539c-9c87-4c00-bca4-91a544602427"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:43:42.726515 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.726484    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4903539c-9c87-4c00-bca4-91a544602427-kube-api-access-xmjnc" (OuterVolumeSpecName: "kube-api-access-xmjnc") pod "4903539c-9c87-4c00-bca4-91a544602427" (UID: "4903539c-9c87-4c00-bca4-91a544602427"). InnerVolumeSpecName "kube-api-access-xmjnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:43:42.726515 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.726502    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ebc494-ce47-46e5-86f8-eccdcab94f58-kube-api-access-qcfhs" (OuterVolumeSpecName: "kube-api-access-qcfhs") pod "42ebc494-ce47-46e5-86f8-eccdcab94f58" (UID: "42ebc494-ce47-46e5-86f8-eccdcab94f58"). InnerVolumeSpecName "kube-api-access-qcfhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:43:42.806762 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.806722    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"
Sep 04 23:43:42.815597 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.815571    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-m4dvc"
Sep 04 23:43:42.825245 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.825224    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4903539c-9c87-4c00-bca4-91a544602427-extensions-socket-volume\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:43:42.825295 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.825248    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmjnc\" (UniqueName: \"kubernetes.io/projected/4903539c-9c87-4c00-bca4-91a544602427-kube-api-access-xmjnc\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:43:42.825295 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:42.825258    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcfhs\" (UniqueName: \"kubernetes.io/projected/42ebc494-ce47-46e5-86f8-eccdcab94f58-kube-api-access-qcfhs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:43:43.087002 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.086880    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"]
Sep 04 23:43:43.090006 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:43:43.089983    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37056ead_63b9_498f_8622_feb8fd337188.slice/crio-a79529337b96f13bdf254cc3302d9d0290fc7922bfade96c0d0300da66b6bbff WatchSource:0}: Error finding container a79529337b96f13bdf254cc3302d9d0290fc7922bfade96c0d0300da66b6bbff: Status 404 returned error can't find the container with id a79529337b96f13bdf254cc3302d9d0290fc7922bfade96c0d0300da66b6bbff
Sep 04 23:43:43.099400 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.099379    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-m4dvc"]
Sep 04 23:43:43.100795 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:43:43.100771    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632a7bdc_5cb2_453c_87fb_72557a347170.slice/crio-a66ea1c3c1209e9c83cc91497d05d0a1935ef0bf88c6281e0a5bc5df4cf09723 WatchSource:0}: Error finding container a66ea1c3c1209e9c83cc91497d05d0a1935ef0bf88c6281e0a5bc5df4cf09723: Status 404 returned error can't find the container with id a66ea1c3c1209e9c83cc91497d05d0a1935ef0bf88c6281e0a5bc5df4cf09723
Sep 04 23:43:43.444093 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.444060    2562 generic.go:356] "Generic (PLEG): container finished" podID="4903539c-9c87-4c00-bca4-91a544602427" containerID="2f5f63747b318babdc96af2a168d23e94d19eba743ecbfe2b56299a118a359fe" exitCode=0
Sep 04 23:43:43.444487 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.444100    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:43.444487 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.444152    2562 scope.go:117] "RemoveContainer" containerID="2f5f63747b318babdc96af2a168d23e94d19eba743ecbfe2b56299a118a359fe"
Sep 04 23:43:43.445618 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.445545    2562 generic.go:356] "Generic (PLEG): container finished" podID="42ebc494-ce47-46e5-86f8-eccdcab94f58" containerID="defdc5a7958c1fbae27be55b4133cc0dde6c1a3f5091bef95d63b6b6489a1d94" exitCode=0
Sep 04 23:43:43.445618 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.445585    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:43.448917 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:43.448306    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zm7pm\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="42ebc494-ce47-46e5-86f8-eccdcab94f58" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:43.450601 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:43.450343    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="4903539c-9c87-4c00-bca4-91a544602427" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:43.452342 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:43.452303    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zm7pm\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="42ebc494-ce47-46e5-86f8-eccdcab94f58" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:43.454319 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:43.454147    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="4903539c-9c87-4c00-bca4-91a544602427" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:43.456567 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.456372    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-m4dvc" event={"ID":"632a7bdc-5cb2-453c-87fb-72557a347170","Type":"ContainerStarted","Data":"f03a2434e5b169e47b5fb42ed9a5cdd26885ddf7ddfdc1182f1a27b112a34db2"}
Sep 04 23:43:43.456567 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.456452    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-m4dvc" event={"ID":"632a7bdc-5cb2-453c-87fb-72557a347170","Type":"ContainerStarted","Data":"a66ea1c3c1209e9c83cc91497d05d0a1935ef0bf88c6281e0a5bc5df4cf09723"}
Sep 04 23:43:43.456567 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.456564    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-m4dvc"
Sep 04 23:43:43.458474 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:43.458417    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zm7pm\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="42ebc494-ce47-46e5-86f8-eccdcab94f58" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:43.459002 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.458980    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk" event={"ID":"37056ead-63b9-498f-8622-feb8fd337188","Type":"ContainerStarted","Data":"e26e1f17e9c44237d75e7c7209439393106c18734081a0f51e0aab6bb054bbb6"}
Sep 04 23:43:43.459086 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.459017    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk" event={"ID":"37056ead-63b9-498f-8622-feb8fd337188","Type":"ContainerStarted","Data":"a79529337b96f13bdf254cc3302d9d0290fc7922bfade96c0d0300da66b6bbff"}
Sep 04 23:43:43.459232 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.459092    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"
Sep 04 23:43:43.459232 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.459206    2562 scope.go:117] "RemoveContainer" containerID="2f5f63747b318babdc96af2a168d23e94d19eba743ecbfe2b56299a118a359fe"
Sep 04 23:43:43.459497 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:43.459475    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5f63747b318babdc96af2a168d23e94d19eba743ecbfe2b56299a118a359fe\": container with ID starting with 2f5f63747b318babdc96af2a168d23e94d19eba743ecbfe2b56299a118a359fe not found: ID does not exist" containerID="2f5f63747b318babdc96af2a168d23e94d19eba743ecbfe2b56299a118a359fe"
Sep 04 23:43:43.459563 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.459506    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5f63747b318babdc96af2a168d23e94d19eba743ecbfe2b56299a118a359fe"} err="failed to get container status \"2f5f63747b318babdc96af2a168d23e94d19eba743ecbfe2b56299a118a359fe\": rpc error: code = NotFound desc = could not find container \"2f5f63747b318babdc96af2a168d23e94d19eba743ecbfe2b56299a118a359fe\": container with ID starting with 2f5f63747b318babdc96af2a168d23e94d19eba743ecbfe2b56299a118a359fe not found: ID does not exist"
Sep 04 23:43:43.459563 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.459528    2562 scope.go:117] "RemoveContainer" containerID="defdc5a7958c1fbae27be55b4133cc0dde6c1a3f5091bef95d63b6b6489a1d94"
Sep 04 23:43:43.467163 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.467147    2562 scope.go:117] "RemoveContainer" containerID="defdc5a7958c1fbae27be55b4133cc0dde6c1a3f5091bef95d63b6b6489a1d94"
Sep 04 23:43:43.467412 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:43.467393    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"defdc5a7958c1fbae27be55b4133cc0dde6c1a3f5091bef95d63b6b6489a1d94\": container with ID starting with defdc5a7958c1fbae27be55b4133cc0dde6c1a3f5091bef95d63b6b6489a1d94 not found: ID does not exist" containerID="defdc5a7958c1fbae27be55b4133cc0dde6c1a3f5091bef95d63b6b6489a1d94"
Sep 04 23:43:43.467462 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.467421    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"defdc5a7958c1fbae27be55b4133cc0dde6c1a3f5091bef95d63b6b6489a1d94"} err="failed to get container status \"defdc5a7958c1fbae27be55b4133cc0dde6c1a3f5091bef95d63b6b6489a1d94\": rpc error: code = NotFound desc = could not find container \"defdc5a7958c1fbae27be55b4133cc0dde6c1a3f5091bef95d63b6b6489a1d94\": container with ID starting with defdc5a7958c1fbae27be55b4133cc0dde6c1a3f5091bef95d63b6b6489a1d94 not found: ID does not exist"
Sep 04 23:43:43.476585 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.476552    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-m4dvc" podStartSLOduration=1.476541127 podStartE2EDuration="1.476541127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:43:43.475227653 +0000 UTC m=+435.721894744" watchObservedRunningTime="2026-09-04 23:43:43.476541127 +0000 UTC m=+435.723208205"
Sep 04 23:43:43.477210 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:43.477189    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="4903539c-9c87-4c00-bca4-91a544602427" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:43.478998 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:43.478978    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-s6n6s\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="4903539c-9c87-4c00-bca4-91a544602427" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s6n6s"
Sep 04 23:43:43.480600 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:43.480584    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zm7pm\" is forbidden: User \"system:node:ip-10-0-138-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-66.ec2.internal' and this object" podUID="42ebc494-ce47-46e5-86f8-eccdcab94f58" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zm7pm"
Sep 04 23:43:43.499738 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:43.499694    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk" podStartSLOduration=1.499681474 podStartE2EDuration="1.499681474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:43:43.498350725 +0000 UTC m=+435.745017801" watchObservedRunningTime="2026-09-04 23:43:43.499681474 +0000 UTC m=+435.746348544"
Sep 04 23:43:44.358333 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:44.358302    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42ebc494-ce47-46e5-86f8-eccdcab94f58" path="/var/lib/kubelet/pods/42ebc494-ce47-46e5-86f8-eccdcab94f58/volumes"
Sep 04 23:43:44.358707 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:44.358693    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4903539c-9c87-4c00-bca4-91a544602427" path="/var/lib/kubelet/pods/4903539c-9c87-4c00-bca4-91a544602427/volumes"
Sep 04 23:43:45.401283 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:45.401251    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-gl4w2"
Sep 04 23:43:54.468588 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:54.468555    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-m4dvc"
Sep 04 23:43:54.468998 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:54.468628    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"
Sep 04 23:43:59.064356 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.064318    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"]
Sep 04 23:43:59.064882 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.064559    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk" podUID="37056ead-63b9-498f-8622-feb8fd337188" containerName="manager" containerID="cri-o://e26e1f17e9c44237d75e7c7209439393106c18734081a0f51e0aab6bb054bbb6" gracePeriod=10
Sep 04 23:43:59.311738 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.311709    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"
Sep 04 23:43:59.362952 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.362861    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37056ead-63b9-498f-8622-feb8fd337188-extensions-socket-volume\") pod \"37056ead-63b9-498f-8622-feb8fd337188\" (UID: \"37056ead-63b9-498f-8622-feb8fd337188\") "
Sep 04 23:43:59.363124 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.363016    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vzph\" (UniqueName: \"kubernetes.io/projected/37056ead-63b9-498f-8622-feb8fd337188-kube-api-access-8vzph\") pod \"37056ead-63b9-498f-8622-feb8fd337188\" (UID: \"37056ead-63b9-498f-8622-feb8fd337188\") "
Sep 04 23:43:59.363247 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.363217    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37056ead-63b9-498f-8622-feb8fd337188-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "37056ead-63b9-498f-8622-feb8fd337188" (UID: "37056ead-63b9-498f-8622-feb8fd337188"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:43:59.365082 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.365058    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37056ead-63b9-498f-8622-feb8fd337188-kube-api-access-8vzph" (OuterVolumeSpecName: "kube-api-access-8vzph") pod "37056ead-63b9-498f-8622-feb8fd337188" (UID: "37056ead-63b9-498f-8622-feb8fd337188"). InnerVolumeSpecName "kube-api-access-8vzph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:43:59.463839 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.463803    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vzph\" (UniqueName: \"kubernetes.io/projected/37056ead-63b9-498f-8622-feb8fd337188-kube-api-access-8vzph\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:43:59.463839 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.463832    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37056ead-63b9-498f-8622-feb8fd337188-extensions-socket-volume\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:43:59.539847 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.539812    2562 generic.go:356] "Generic (PLEG): container finished" podID="37056ead-63b9-498f-8622-feb8fd337188" containerID="e26e1f17e9c44237d75e7c7209439393106c18734081a0f51e0aab6bb054bbb6" exitCode=0
Sep 04 23:43:59.540024 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.539865    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"
Sep 04 23:43:59.540024 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.539898    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk" event={"ID":"37056ead-63b9-498f-8622-feb8fd337188","Type":"ContainerDied","Data":"e26e1f17e9c44237d75e7c7209439393106c18734081a0f51e0aab6bb054bbb6"}
Sep 04 23:43:59.540024 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.539955    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk" event={"ID":"37056ead-63b9-498f-8622-feb8fd337188","Type":"ContainerDied","Data":"a79529337b96f13bdf254cc3302d9d0290fc7922bfade96c0d0300da66b6bbff"}
Sep 04 23:43:59.540024 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.539978    2562 scope.go:117] "RemoveContainer" containerID="e26e1f17e9c44237d75e7c7209439393106c18734081a0f51e0aab6bb054bbb6"
Sep 04 23:43:59.548509 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.548492    2562 scope.go:117] "RemoveContainer" containerID="e26e1f17e9c44237d75e7c7209439393106c18734081a0f51e0aab6bb054bbb6"
Sep 04 23:43:59.548822 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:43:59.548796    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26e1f17e9c44237d75e7c7209439393106c18734081a0f51e0aab6bb054bbb6\": container with ID starting with e26e1f17e9c44237d75e7c7209439393106c18734081a0f51e0aab6bb054bbb6 not found: ID does not exist" containerID="e26e1f17e9c44237d75e7c7209439393106c18734081a0f51e0aab6bb054bbb6"
Sep 04 23:43:59.548896 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.548833    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26e1f17e9c44237d75e7c7209439393106c18734081a0f51e0aab6bb054bbb6"} err="failed to get container status \"e26e1f17e9c44237d75e7c7209439393106c18734081a0f51e0aab6bb054bbb6\": rpc error: code = NotFound desc = could not find container \"e26e1f17e9c44237d75e7c7209439393106c18734081a0f51e0aab6bb054bbb6\": container with ID starting with e26e1f17e9c44237d75e7c7209439393106c18734081a0f51e0aab6bb054bbb6 not found: ID does not exist"
Sep 04 23:43:59.563563 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.563543    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"]
Sep 04 23:43:59.570375 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:43:59.570354    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-hnpqk"]
Sep 04 23:44:00.358565 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:00.358532    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37056ead-63b9-498f-8622-feb8fd337188" path="/var/lib/kubelet/pods/37056ead-63b9-498f-8622-feb8fd337188/volumes"
Sep 04 23:44:15.309317 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.309279    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"]
Sep 04 23:44:15.309893 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.309875    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37056ead-63b9-498f-8622-feb8fd337188" containerName="manager"
Sep 04 23:44:15.309937 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.309898    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="37056ead-63b9-498f-8622-feb8fd337188" containerName="manager"
Sep 04 23:44:15.310002 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.309991    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="37056ead-63b9-498f-8622-feb8fd337188" containerName="manager"
Sep 04 23:44:15.327896 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.327870    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"]
Sep 04 23:44:15.328043 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.327979    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.330280 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.330254    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9zmcd\""
Sep 04 23:44:15.330413 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.330261    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 04 23:44:15.399447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.399407    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9907f4b-6c2c-4b46-b927-ff56ad289e45-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.399639 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.399455    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9907f4b-6c2c-4b46-b927-ff56ad289e45-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.399639 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.399511    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9907f4b-6c2c-4b46-b927-ff56ad289e45-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.399639 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.399554    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9907f4b-6c2c-4b46-b927-ff56ad289e45-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.399639 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.399580    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9907f4b-6c2c-4b46-b927-ff56ad289e45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.399639 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.399607    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9907f4b-6c2c-4b46-b927-ff56ad289e45-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.399902 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.399741    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9907f4b-6c2c-4b46-b927-ff56ad289e45-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.399902 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.399776    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9907f4b-6c2c-4b46-b927-ff56ad289e45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.399902 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.399811    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6rcf\" (UniqueName: \"kubernetes.io/projected/d9907f4b-6c2c-4b46-b927-ff56ad289e45-kube-api-access-p6rcf\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.500465 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.500434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9907f4b-6c2c-4b46-b927-ff56ad289e45-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.500465 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.500468    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9907f4b-6c2c-4b46-b927-ff56ad289e45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.500726 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.500504    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6rcf\" (UniqueName: \"kubernetes.io/projected/d9907f4b-6c2c-4b46-b927-ff56ad289e45-kube-api-access-p6rcf\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.500790 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.500717    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9907f4b-6c2c-4b46-b927-ff56ad289e45-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.500790 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.500766    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9907f4b-6c2c-4b46-b927-ff56ad289e45-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.500932 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.500911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9907f4b-6c2c-4b46-b927-ff56ad289e45-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.501005 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.500956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9907f4b-6c2c-4b46-b927-ff56ad289e45-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.501005 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.500986    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9907f4b-6c2c-4b46-b927-ff56ad289e45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.501108 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.501017    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9907f4b-6c2c-4b46-b927-ff56ad289e45-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.501108 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.501074    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9907f4b-6c2c-4b46-b927-ff56ad289e45-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.501322 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.501293    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9907f4b-6c2c-4b46-b927-ff56ad289e45-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.501387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.501370    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9907f4b-6c2c-4b46-b927-ff56ad289e45-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.501476 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.501453    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9907f4b-6c2c-4b46-b927-ff56ad289e45-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.501794 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.501770    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9907f4b-6c2c-4b46-b927-ff56ad289e45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.502923 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.502902    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9907f4b-6c2c-4b46-b927-ff56ad289e45-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.503255 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.503235    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9907f4b-6c2c-4b46-b927-ff56ad289e45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.508151 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.508122    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9907f4b-6c2c-4b46-b927-ff56ad289e45-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.508253 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.508164    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6rcf\" (UniqueName: \"kubernetes.io/projected/d9907f4b-6c2c-4b46-b927-ff56ad289e45-kube-api-access-p6rcf\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ln8g4\" (UID: \"d9907f4b-6c2c-4b46-b927-ff56ad289e45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.638604 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.638533    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:15.912432 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:15.912349    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"]
Sep 04 23:44:15.915709 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:44:15.915678    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9907f4b_6c2c_4b46_b927_ff56ad289e45.slice/crio-132c03156ca48c8eb8e5d7a12f0deefa67c0667133b1ff6caedf013bdd51c764 WatchSource:0}: Error finding container 132c03156ca48c8eb8e5d7a12f0deefa67c0667133b1ff6caedf013bdd51c764: Status 404 returned error can't find the container with id 132c03156ca48c8eb8e5d7a12f0deefa67c0667133b1ff6caedf013bdd51c764
Sep 04 23:44:16.623044 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:16.622971    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4" event={"ID":"d9907f4b-6c2c-4b46-b927-ff56ad289e45","Type":"ContainerStarted","Data":"132c03156ca48c8eb8e5d7a12f0deefa67c0667133b1ff6caedf013bdd51c764"}
Sep 04 23:44:18.380548 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:18.380515    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:44:18.380846 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:18.380579    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:44:18.380846 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:18.380610    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:44:18.635012 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:18.634921    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4" event={"ID":"d9907f4b-6c2c-4b46-b927-ff56ad289e45","Type":"ContainerStarted","Data":"fdda3df137eacf2dc9df8ab37375225f0d227171e507fc14e832d544b2abe6b5"}
Sep 04 23:44:18.638935 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:18.638913    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:18.655011 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:18.654964    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4" podStartSLOduration=1.192496999 podStartE2EDuration="3.654948474s" totalImagesPullingTime="2.462451475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:44:18.652842604 +0000 UTC m=+470.899509685" watchObservedRunningTime="2026-09-04 23:44:18.654948474 +0000 UTC m=+470.901615552"
Sep 04 23:44:19.634926 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:19.634891    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"]
Sep 04 23:44:19.637890 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:19.637867    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"
Sep 04 23:44:19.640313 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:19.640286    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 04 23:44:19.643438 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:19.643414    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:19.645724 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:19.645703    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"]
Sep 04 23:44:19.737263 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:19.737232    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"]
Sep 04 23:44:19.740089 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:19.740063    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31e71151-7bfc-41fe-85c9-6ab00c5483ef-config-file\") pod \"limitador-limitador-859c4d96d6-6rhkv\" (UID: \"31e71151-7bfc-41fe-85c9-6ab00c5483ef\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"
Sep 04 23:44:19.740311 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:19.740216    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gv7k\" (UniqueName: \"kubernetes.io/projected/31e71151-7bfc-41fe-85c9-6ab00c5483ef-kube-api-access-8gv7k\") pod \"limitador-limitador-859c4d96d6-6rhkv\" (UID: \"31e71151-7bfc-41fe-85c9-6ab00c5483ef\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"
Sep 04 23:44:19.841438 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:19.841401    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gv7k\" (UniqueName: \"kubernetes.io/projected/31e71151-7bfc-41fe-85c9-6ab00c5483ef-kube-api-access-8gv7k\") pod \"limitador-limitador-859c4d96d6-6rhkv\" (UID: \"31e71151-7bfc-41fe-85c9-6ab00c5483ef\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"
Sep 04 23:44:19.841617 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:19.841460    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31e71151-7bfc-41fe-85c9-6ab00c5483ef-config-file\") pod \"limitador-limitador-859c4d96d6-6rhkv\" (UID: \"31e71151-7bfc-41fe-85c9-6ab00c5483ef\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"
Sep 04 23:44:19.842094 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:19.842075    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31e71151-7bfc-41fe-85c9-6ab00c5483ef-config-file\") pod \"limitador-limitador-859c4d96d6-6rhkv\" (UID: \"31e71151-7bfc-41fe-85c9-6ab00c5483ef\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"
Sep 04 23:44:19.848697 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:19.848673    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gv7k\" (UniqueName: \"kubernetes.io/projected/31e71151-7bfc-41fe-85c9-6ab00c5483ef-kube-api-access-8gv7k\") pod \"limitador-limitador-859c4d96d6-6rhkv\" (UID: \"31e71151-7bfc-41fe-85c9-6ab00c5483ef\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"
Sep 04 23:44:19.948868 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:19.948786    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"
Sep 04 23:44:20.216798 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:20.216776    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"]
Sep 04 23:44:20.218885 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:44:20.218853    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e71151_7bfc_41fe_85c9_6ab00c5483ef.slice/crio-d268555c728165255cf53ce8a68c8884aff6797002aaba0cdadd0accb41e72ca WatchSource:0}: Error finding container d268555c728165255cf53ce8a68c8884aff6797002aaba0cdadd0accb41e72ca: Status 404 returned error can't find the container with id d268555c728165255cf53ce8a68c8884aff6797002aaba0cdadd0accb41e72ca
Sep 04 23:44:20.550890 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:20.550816    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-r8f85"]
Sep 04 23:44:20.553790 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:20.553774    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-r8f85"
Sep 04 23:44:20.556473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:20.556446    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rx5zz\""
Sep 04 23:44:20.558860 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:20.558836    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-r8f85"]
Sep 04 23:44:20.645838 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:20.645799    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv" event={"ID":"31e71151-7bfc-41fe-85c9-6ab00c5483ef","Type":"ContainerStarted","Data":"d268555c728165255cf53ce8a68c8884aff6797002aaba0cdadd0accb41e72ca"}
Sep 04 23:44:20.646214 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:20.645985    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:20.647147 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:20.647127    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:44:20.648422 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:20.648403    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffzn9\" (UniqueName: \"kubernetes.io/projected/0d597c6e-2184-4cf6-8892-09444af5226e-kube-api-access-ffzn9\") pod \"authorino-c548b5d7d-r8f85\" (UID: \"0d597c6e-2184-4cf6-8892-09444af5226e\") " pod="kuadrant-system/authorino-c548b5d7d-r8f85"
Sep 04 23:44:20.749774 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:20.749736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffzn9\" (UniqueName: \"kubernetes.io/projected/0d597c6e-2184-4cf6-8892-09444af5226e-kube-api-access-ffzn9\") pod \"authorino-c548b5d7d-r8f85\" (UID: \"0d597c6e-2184-4cf6-8892-09444af5226e\") " pod="kuadrant-system/authorino-c548b5d7d-r8f85"
Sep 04 23:44:20.761354 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:20.761328    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffzn9\" (UniqueName: \"kubernetes.io/projected/0d597c6e-2184-4cf6-8892-09444af5226e-kube-api-access-ffzn9\") pod \"authorino-c548b5d7d-r8f85\" (UID: \"0d597c6e-2184-4cf6-8892-09444af5226e\") " pod="kuadrant-system/authorino-c548b5d7d-r8f85"
Sep 04 23:44:20.863936 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:20.863856    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-r8f85"
Sep 04 23:44:21.131245 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:21.131164    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-r8f85"]
Sep 04 23:44:21.134824 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:44:21.134796    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d597c6e_2184_4cf6_8892_09444af5226e.slice/crio-a473dc07a8e40118c6af6967ada59eb67ed6ebbf51c37927a0809926bf73b527 WatchSource:0}: Error finding container a473dc07a8e40118c6af6967ada59eb67ed6ebbf51c37927a0809926bf73b527: Status 404 returned error can't find the container with id a473dc07a8e40118c6af6967ada59eb67ed6ebbf51c37927a0809926bf73b527
Sep 04 23:44:21.651915 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:21.651879    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-r8f85" event={"ID":"0d597c6e-2184-4cf6-8892-09444af5226e","Type":"ContainerStarted","Data":"a473dc07a8e40118c6af6967ada59eb67ed6ebbf51c37927a0809926bf73b527"}
Sep 04 23:44:21.653345 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:21.653310    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv" event={"ID":"31e71151-7bfc-41fe-85c9-6ab00c5483ef","Type":"ContainerStarted","Data":"11d8160e036f1bb3918853ec0daa2d334d1537fd821bb53927459f8a05f8b11c"}
Sep 04 23:44:21.653547 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:21.653528    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"
Sep 04 23:44:21.669725 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:21.669622    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv" podStartSLOduration=1.472115334 podStartE2EDuration="2.669606904s" totalImagesPullingTime="1.19749157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:44:21.66844251 +0000 UTC m=+473.915109592" watchObservedRunningTime="2026-09-04 23:44:21.669606904 +0000 UTC m=+473.916273983"
Sep 04 23:44:23.665024 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:23.664993    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-r8f85" event={"ID":"0d597c6e-2184-4cf6-8892-09444af5226e","Type":"ContainerStarted","Data":"09f8d01eebf5289cfb998702bcb8418f38ca4547a7e336b82021dcd065b2f3ce"}
Sep 04 23:44:23.678260 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:23.678209    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-r8f85" podStartSLOduration=1.899062947 podStartE2EDuration="3.678193686s" totalImagesPullingTime="1.779130739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:44:23.677812442 +0000 UTC m=+475.924479524" watchObservedRunningTime="2026-09-04 23:44:23.678193686 +0000 UTC m=+475.924860764"
Sep 04 23:44:32.660560 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:32.660526    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"
Sep 04 23:44:34.143882 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:34.143837    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"]
Sep 04 23:44:34.144338 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:34.144073    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv" podUID="31e71151-7bfc-41fe-85c9-6ab00c5483ef" containerName="limitador" containerID="cri-o://11d8160e036f1bb3918853ec0daa2d334d1537fd821bb53927459f8a05f8b11c" gracePeriod=30
Sep 04 23:44:34.719749 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:34.719638    2562 generic.go:356] "Generic (PLEG): container finished" podID="31e71151-7bfc-41fe-85c9-6ab00c5483ef" containerID="11d8160e036f1bb3918853ec0daa2d334d1537fd821bb53927459f8a05f8b11c" exitCode=0
Sep 04 23:44:34.719749 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:34.719723    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv" event={"ID":"31e71151-7bfc-41fe-85c9-6ab00c5483ef","Type":"ContainerDied","Data":"11d8160e036f1bb3918853ec0daa2d334d1537fd821bb53927459f8a05f8b11c"}
Sep 04 23:44:35.099068 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:35.099041    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"
Sep 04 23:44:35.170770 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:35.170739    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gv7k\" (UniqueName: \"kubernetes.io/projected/31e71151-7bfc-41fe-85c9-6ab00c5483ef-kube-api-access-8gv7k\") pod \"31e71151-7bfc-41fe-85c9-6ab00c5483ef\" (UID: \"31e71151-7bfc-41fe-85c9-6ab00c5483ef\") "
Sep 04 23:44:35.171138 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:35.170803    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31e71151-7bfc-41fe-85c9-6ab00c5483ef-config-file\") pod \"31e71151-7bfc-41fe-85c9-6ab00c5483ef\" (UID: \"31e71151-7bfc-41fe-85c9-6ab00c5483ef\") "
Sep 04 23:44:35.171138 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:35.171110    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e71151-7bfc-41fe-85c9-6ab00c5483ef-config-file" (OuterVolumeSpecName: "config-file") pod "31e71151-7bfc-41fe-85c9-6ab00c5483ef" (UID: "31e71151-7bfc-41fe-85c9-6ab00c5483ef"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:44:35.172764 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:35.172743    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e71151-7bfc-41fe-85c9-6ab00c5483ef-kube-api-access-8gv7k" (OuterVolumeSpecName: "kube-api-access-8gv7k") pod "31e71151-7bfc-41fe-85c9-6ab00c5483ef" (UID: "31e71151-7bfc-41fe-85c9-6ab00c5483ef"). InnerVolumeSpecName "kube-api-access-8gv7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:44:35.272374 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:35.272297    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gv7k\" (UniqueName: \"kubernetes.io/projected/31e71151-7bfc-41fe-85c9-6ab00c5483ef-kube-api-access-8gv7k\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:44:35.272374 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:35.272322    2562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31e71151-7bfc-41fe-85c9-6ab00c5483ef-config-file\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:44:35.725292 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:35.725263    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"
Sep 04 23:44:35.725453 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:35.725299    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-6rhkv" event={"ID":"31e71151-7bfc-41fe-85c9-6ab00c5483ef","Type":"ContainerDied","Data":"d268555c728165255cf53ce8a68c8884aff6797002aaba0cdadd0accb41e72ca"}
Sep 04 23:44:35.725453 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:35.725360    2562 scope.go:117] "RemoveContainer" containerID="11d8160e036f1bb3918853ec0daa2d334d1537fd821bb53927459f8a05f8b11c"
Sep 04 23:44:35.745399 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:35.745370    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"]
Sep 04 23:44:35.749831 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:35.749808    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-6rhkv"]
Sep 04 23:44:36.061465 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.061390    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wth2h"]
Sep 04 23:44:36.061948 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.061929    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31e71151-7bfc-41fe-85c9-6ab00c5483ef" containerName="limitador"
Sep 04 23:44:36.062009 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.061952    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="31e71151-7bfc-41fe-85c9-6ab00c5483ef" containerName="limitador"
Sep 04 23:44:36.062093 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.062081    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="31e71151-7bfc-41fe-85c9-6ab00c5483ef" containerName="limitador"
Sep 04 23:44:36.066442 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.066411    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wth2h"
Sep 04 23:44:36.069163 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.069078    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 23:44:36.069316 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.069175    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 04 23:44:36.070291 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.070121    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-9trjw\""
Sep 04 23:44:36.070291 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.070178    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 04 23:44:36.072196 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.072171    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wth2h"]
Sep 04 23:44:36.080724 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.080695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmnq5\" (UniqueName: \"kubernetes.io/projected/15ebf5ad-5286-4ba7-b5e2-bf78192b5091-kube-api-access-kmnq5\") pod \"postgres-69fb4594cf-wth2h\" (UID: \"15ebf5ad-5286-4ba7-b5e2-bf78192b5091\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wth2h"
Sep 04 23:44:36.080821 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.080758    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/15ebf5ad-5286-4ba7-b5e2-bf78192b5091-data\") pod \"postgres-69fb4594cf-wth2h\" (UID: \"15ebf5ad-5286-4ba7-b5e2-bf78192b5091\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wth2h"
Sep 04 23:44:36.181557 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.181521    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmnq5\" (UniqueName: \"kubernetes.io/projected/15ebf5ad-5286-4ba7-b5e2-bf78192b5091-kube-api-access-kmnq5\") pod \"postgres-69fb4594cf-wth2h\" (UID: \"15ebf5ad-5286-4ba7-b5e2-bf78192b5091\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wth2h"
Sep 04 23:44:36.182029 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.181568    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/15ebf5ad-5286-4ba7-b5e2-bf78192b5091-data\") pod \"postgres-69fb4594cf-wth2h\" (UID: \"15ebf5ad-5286-4ba7-b5e2-bf78192b5091\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wth2h"
Sep 04 23:44:36.182070 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.182039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/15ebf5ad-5286-4ba7-b5e2-bf78192b5091-data\") pod \"postgres-69fb4594cf-wth2h\" (UID: \"15ebf5ad-5286-4ba7-b5e2-bf78192b5091\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wth2h"
Sep 04 23:44:36.189826 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.189799    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmnq5\" (UniqueName: \"kubernetes.io/projected/15ebf5ad-5286-4ba7-b5e2-bf78192b5091-kube-api-access-kmnq5\") pod \"postgres-69fb4594cf-wth2h\" (UID: \"15ebf5ad-5286-4ba7-b5e2-bf78192b5091\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wth2h"
Sep 04 23:44:36.358782 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.358707    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31e71151-7bfc-41fe-85c9-6ab00c5483ef" path="/var/lib/kubelet/pods/31e71151-7bfc-41fe-85c9-6ab00c5483ef/volumes"
Sep 04 23:44:36.377825 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.377804    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wth2h"
Sep 04 23:44:36.648440 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.648367    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wth2h"]
Sep 04 23:44:36.651265 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:44:36.651235    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ebf5ad_5286_4ba7_b5e2_bf78192b5091.slice/crio-e5f55dee036c51278fa736b6c7140ab5b397cf366f44aa1883af88402bc086a0 WatchSource:0}: Error finding container e5f55dee036c51278fa736b6c7140ab5b397cf366f44aa1883af88402bc086a0: Status 404 returned error can't find the container with id e5f55dee036c51278fa736b6c7140ab5b397cf366f44aa1883af88402bc086a0
Sep 04 23:44:36.732195 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:36.732158    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wth2h" event={"ID":"15ebf5ad-5286-4ba7-b5e2-bf78192b5091","Type":"ContainerStarted","Data":"e5f55dee036c51278fa736b6c7140ab5b397cf366f44aa1883af88402bc086a0"}
Sep 04 23:44:42.770562 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:42.770527    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wth2h" event={"ID":"15ebf5ad-5286-4ba7-b5e2-bf78192b5091","Type":"ContainerStarted","Data":"d8120accf878813860508aa497ce8cec9504583b293839d0f8028d1e1e09979b"}
Sep 04 23:44:42.770955 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:42.770635    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wth2h"
Sep 04 23:44:42.785042 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:42.784978    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wth2h" podStartSLOduration=1.492510186 podStartE2EDuration="6.784964592s" totalImagesPullingTime="5.292454406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:44:42.784138364 +0000 UTC m=+495.030805443" watchObservedRunningTime="2026-09-04 23:44:42.784964592 +0000 UTC m=+495.031631678"
Sep 04 23:44:48.804809 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:48.804781    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wth2h"
Sep 04 23:44:49.311909 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.311878    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-45qnz"]
Sep 04 23:44:49.317191 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.317169    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-45qnz"
Sep 04 23:44:49.321090 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.321064    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-45qnz"]
Sep 04 23:44:49.371703 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.371633    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-45qnz"]
Sep 04 23:44:49.372361 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:44:49.372291    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-d7jzq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-45qnz" podUID="3544fe45-2a19-4618-85ce-706c3c3d66a8"
Sep 04 23:44:49.397120 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.397091    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-582hg"]
Sep 04 23:44:49.400573 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.400555    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-582hg"
Sep 04 23:44:49.402292 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.402268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7jzq\" (UniqueName: \"kubernetes.io/projected/3544fe45-2a19-4618-85ce-706c3c3d66a8-kube-api-access-d7jzq\") pod \"authorino-5dc7ccb5f-45qnz\" (UID: \"3544fe45-2a19-4618-85ce-706c3c3d66a8\") " pod="kuadrant-system/authorino-5dc7ccb5f-45qnz"
Sep 04 23:44:49.402860 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.402839    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 04 23:44:49.408690 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.408666    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-582hg"]
Sep 04 23:44:49.503565 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.503526    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jzq\" (UniqueName: \"kubernetes.io/projected/3544fe45-2a19-4618-85ce-706c3c3d66a8-kube-api-access-d7jzq\") pod \"authorino-5dc7ccb5f-45qnz\" (UID: \"3544fe45-2a19-4618-85ce-706c3c3d66a8\") " pod="kuadrant-system/authorino-5dc7ccb5f-45qnz"
Sep 04 23:44:49.503786 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.503572    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rcsc\" (UniqueName: \"kubernetes.io/projected/5d6d8d01-1bc0-4128-b3d4-7155501bd8ba-kube-api-access-5rcsc\") pod \"authorino-687bd45585-582hg\" (UID: \"5d6d8d01-1bc0-4128-b3d4-7155501bd8ba\") " pod="kuadrant-system/authorino-687bd45585-582hg"
Sep 04 23:44:49.503786 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.503630    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d6d8d01-1bc0-4128-b3d4-7155501bd8ba-tls-cert\") pod \"authorino-687bd45585-582hg\" (UID: \"5d6d8d01-1bc0-4128-b3d4-7155501bd8ba\") " pod="kuadrant-system/authorino-687bd45585-582hg"
Sep 04 23:44:49.511393 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.511364    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7jzq\" (UniqueName: \"kubernetes.io/projected/3544fe45-2a19-4618-85ce-706c3c3d66a8-kube-api-access-d7jzq\") pod \"authorino-5dc7ccb5f-45qnz\" (UID: \"3544fe45-2a19-4618-85ce-706c3c3d66a8\") " pod="kuadrant-system/authorino-5dc7ccb5f-45qnz"
Sep 04 23:44:49.543442 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.543410    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-582hg"]
Sep 04 23:44:49.543686 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:44:49.543669    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5rcsc tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-582hg" podUID="5d6d8d01-1bc0-4128-b3d4-7155501bd8ba"
Sep 04 23:44:49.605055 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.604980    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rcsc\" (UniqueName: \"kubernetes.io/projected/5d6d8d01-1bc0-4128-b3d4-7155501bd8ba-kube-api-access-5rcsc\") pod \"authorino-687bd45585-582hg\" (UID: \"5d6d8d01-1bc0-4128-b3d4-7155501bd8ba\") " pod="kuadrant-system/authorino-687bd45585-582hg"
Sep 04 23:44:49.605055 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.605026    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d6d8d01-1bc0-4128-b3d4-7155501bd8ba-tls-cert\") pod \"authorino-687bd45585-582hg\" (UID: \"5d6d8d01-1bc0-4128-b3d4-7155501bd8ba\") " pod="kuadrant-system/authorino-687bd45585-582hg"
Sep 04 23:44:49.607563 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.607532    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d6d8d01-1bc0-4128-b3d4-7155501bd8ba-tls-cert\") pod \"authorino-687bd45585-582hg\" (UID: \"5d6d8d01-1bc0-4128-b3d4-7155501bd8ba\") " pod="kuadrant-system/authorino-687bd45585-582hg"
Sep 04 23:44:49.612138 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.612116    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rcsc\" (UniqueName: \"kubernetes.io/projected/5d6d8d01-1bc0-4128-b3d4-7155501bd8ba-kube-api-access-5rcsc\") pod \"authorino-687bd45585-582hg\" (UID: \"5d6d8d01-1bc0-4128-b3d4-7155501bd8ba\") " pod="kuadrant-system/authorino-687bd45585-582hg"
Sep 04 23:44:49.802889 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.802860    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-582hg"
Sep 04 23:44:49.803064 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.802874    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-45qnz"
Sep 04 23:44:49.810268 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.810248    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-582hg"
Sep 04 23:44:49.812995 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.812978    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-45qnz"
Sep 04 23:44:49.906701 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.906592    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d6d8d01-1bc0-4128-b3d4-7155501bd8ba-tls-cert\") pod \"5d6d8d01-1bc0-4128-b3d4-7155501bd8ba\" (UID: \"5d6d8d01-1bc0-4128-b3d4-7155501bd8ba\") "
Sep 04 23:44:49.906701 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.906684    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rcsc\" (UniqueName: \"kubernetes.io/projected/5d6d8d01-1bc0-4128-b3d4-7155501bd8ba-kube-api-access-5rcsc\") pod \"5d6d8d01-1bc0-4128-b3d4-7155501bd8ba\" (UID: \"5d6d8d01-1bc0-4128-b3d4-7155501bd8ba\") "
Sep 04 23:44:49.906895 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.906715    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7jzq\" (UniqueName: \"kubernetes.io/projected/3544fe45-2a19-4618-85ce-706c3c3d66a8-kube-api-access-d7jzq\") pod \"3544fe45-2a19-4618-85ce-706c3c3d66a8\" (UID: \"3544fe45-2a19-4618-85ce-706c3c3d66a8\") "
Sep 04 23:44:49.908909 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.908874    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d6d8d01-1bc0-4128-b3d4-7155501bd8ba-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5d6d8d01-1bc0-4128-b3d4-7155501bd8ba" (UID: "5d6d8d01-1bc0-4128-b3d4-7155501bd8ba"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:44:49.909024 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.908969    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3544fe45-2a19-4618-85ce-706c3c3d66a8-kube-api-access-d7jzq" (OuterVolumeSpecName: "kube-api-access-d7jzq") pod "3544fe45-2a19-4618-85ce-706c3c3d66a8" (UID: "3544fe45-2a19-4618-85ce-706c3c3d66a8"). InnerVolumeSpecName "kube-api-access-d7jzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:44:49.909024 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:49.908966    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6d8d01-1bc0-4128-b3d4-7155501bd8ba-kube-api-access-5rcsc" (OuterVolumeSpecName: "kube-api-access-5rcsc") pod "5d6d8d01-1bc0-4128-b3d4-7155501bd8ba" (UID: "5d6d8d01-1bc0-4128-b3d4-7155501bd8ba"). InnerVolumeSpecName "kube-api-access-5rcsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:44:50.008101 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:50.008062    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rcsc\" (UniqueName: \"kubernetes.io/projected/5d6d8d01-1bc0-4128-b3d4-7155501bd8ba-kube-api-access-5rcsc\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:44:50.008101 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:50.008097    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7jzq\" (UniqueName: \"kubernetes.io/projected/3544fe45-2a19-4618-85ce-706c3c3d66a8-kube-api-access-d7jzq\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:44:50.008101 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:50.008109    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d6d8d01-1bc0-4128-b3d4-7155501bd8ba-tls-cert\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:44:50.807144 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:50.807112    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-45qnz"
Sep 04 23:44:50.807319 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:50.807112    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-582hg"
Sep 04 23:44:50.834262 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:50.834233    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-582hg"]
Sep 04 23:44:50.837962 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:50.837936    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-582hg"]
Sep 04 23:44:50.855539 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:50.855514    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-45qnz"]
Sep 04 23:44:50.857820 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:50.857785    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-45qnz"]
Sep 04 23:44:51.322866 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:51.322838    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-r8f85"]
Sep 04 23:44:51.323014 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:51.322999    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-r8f85" podUID="0d597c6e-2184-4cf6-8892-09444af5226e" containerName="authorino" containerID="cri-o://09f8d01eebf5289cfb998702bcb8418f38ca4547a7e336b82021dcd065b2f3ce" gracePeriod=30
Sep 04 23:44:51.565222 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:51.565196    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-r8f85"
Sep 04 23:44:51.621933 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:51.621846    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffzn9\" (UniqueName: \"kubernetes.io/projected/0d597c6e-2184-4cf6-8892-09444af5226e-kube-api-access-ffzn9\") pod \"0d597c6e-2184-4cf6-8892-09444af5226e\" (UID: \"0d597c6e-2184-4cf6-8892-09444af5226e\") "
Sep 04 23:44:51.623973 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:51.623945    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d597c6e-2184-4cf6-8892-09444af5226e-kube-api-access-ffzn9" (OuterVolumeSpecName: "kube-api-access-ffzn9") pod "0d597c6e-2184-4cf6-8892-09444af5226e" (UID: "0d597c6e-2184-4cf6-8892-09444af5226e"). InnerVolumeSpecName "kube-api-access-ffzn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:44:51.722978 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:51.722943    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffzn9\" (UniqueName: \"kubernetes.io/projected/0d597c6e-2184-4cf6-8892-09444af5226e-kube-api-access-ffzn9\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:44:51.813596 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:51.813564    2562 generic.go:356] "Generic (PLEG): container finished" podID="0d597c6e-2184-4cf6-8892-09444af5226e" containerID="09f8d01eebf5289cfb998702bcb8418f38ca4547a7e336b82021dcd065b2f3ce" exitCode=0
Sep 04 23:44:51.813792 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:51.813608    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-r8f85"
Sep 04 23:44:51.813792 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:51.813669    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-r8f85" event={"ID":"0d597c6e-2184-4cf6-8892-09444af5226e","Type":"ContainerDied","Data":"09f8d01eebf5289cfb998702bcb8418f38ca4547a7e336b82021dcd065b2f3ce"}
Sep 04 23:44:51.813792 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:51.813716    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-r8f85" event={"ID":"0d597c6e-2184-4cf6-8892-09444af5226e","Type":"ContainerDied","Data":"a473dc07a8e40118c6af6967ada59eb67ed6ebbf51c37927a0809926bf73b527"}
Sep 04 23:44:51.813792 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:51.813740    2562 scope.go:117] "RemoveContainer" containerID="09f8d01eebf5289cfb998702bcb8418f38ca4547a7e336b82021dcd065b2f3ce"
Sep 04 23:44:51.822591 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:51.822577    2562 scope.go:117] "RemoveContainer" containerID="09f8d01eebf5289cfb998702bcb8418f38ca4547a7e336b82021dcd065b2f3ce"
Sep 04 23:44:51.822869 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:44:51.822852    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f8d01eebf5289cfb998702bcb8418f38ca4547a7e336b82021dcd065b2f3ce\": container with ID starting with 09f8d01eebf5289cfb998702bcb8418f38ca4547a7e336b82021dcd065b2f3ce not found: ID does not exist" containerID="09f8d01eebf5289cfb998702bcb8418f38ca4547a7e336b82021dcd065b2f3ce"
Sep 04 23:44:51.822915 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:51.822877    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f8d01eebf5289cfb998702bcb8418f38ca4547a7e336b82021dcd065b2f3ce"} err="failed to get container status \"09f8d01eebf5289cfb998702bcb8418f38ca4547a7e336b82021dcd065b2f3ce\": rpc error: code = NotFound desc = could not find container \"09f8d01eebf5289cfb998702bcb8418f38ca4547a7e336b82021dcd065b2f3ce\": container with ID starting with 09f8d01eebf5289cfb998702bcb8418f38ca4547a7e336b82021dcd065b2f3ce not found: ID does not exist"
Sep 04 23:44:51.836693 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:51.836672    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-r8f85"]
Sep 04 23:44:51.846816 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:51.846786    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-r8f85"]
Sep 04 23:44:52.358849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:52.358817    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d597c6e-2184-4cf6-8892-09444af5226e" path="/var/lib/kubelet/pods/0d597c6e-2184-4cf6-8892-09444af5226e/volumes"
Sep 04 23:44:52.359144 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:52.359131    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3544fe45-2a19-4618-85ce-706c3c3d66a8" path="/var/lib/kubelet/pods/3544fe45-2a19-4618-85ce-706c3c3d66a8/volumes"
Sep 04 23:44:52.359360 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:52.359343    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d6d8d01-1bc0-4128-b3d4-7155501bd8ba" path="/var/lib/kubelet/pods/5d6d8d01-1bc0-4128-b3d4-7155501bd8ba/volumes"
Sep 04 23:44:55.820997 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.820966    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f664f4546-ck9lx"]
Sep 04 23:44:55.821448 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.821325    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d597c6e-2184-4cf6-8892-09444af5226e" containerName="authorino"
Sep 04 23:44:55.821448 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.821336    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d597c6e-2184-4cf6-8892-09444af5226e" containerName="authorino"
Sep 04 23:44:55.821448 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.821391    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d597c6e-2184-4cf6-8892-09444af5226e" containerName="authorino"
Sep 04 23:44:55.825268 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.825245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:44:55.827771 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.827744    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 04 23:44:55.827891 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.827830    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 04 23:44:55.828590 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.828573    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 04 23:44:55.828702 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.828601    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-llhsg\""
Sep 04 23:44:55.834204 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.834180    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f664f4546-ck9lx"]
Sep 04 23:44:55.958641 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.958603    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64v8b\" (UniqueName: \"kubernetes.io/projected/8e60152e-fb27-4d18-991f-88975edfb782-kube-api-access-64v8b\") pod \"maas-controller-f664f4546-ck9lx\" (UID: \"8e60152e-fb27-4d18-991f-88975edfb782\") " pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:44:55.958830 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.958686    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-webhook-cert\") pod \"maas-controller-f664f4546-ck9lx\" (UID: \"8e60152e-fb27-4d18-991f-88975edfb782\") " pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:44:55.958830 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.958810    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-metrics-certs\") pod \"maas-controller-f664f4546-ck9lx\" (UID: \"8e60152e-fb27-4d18-991f-88975edfb782\") " pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:44:55.984873 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.984848    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64c89c545-phszt"]
Sep 04 23:44:55.987341 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.987320    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 04 23:44:55.996744 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:55.996723    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64c89c545-phszt"]
Sep 04 23:44:56.059976 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.059941    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-webhook-cert\") pod \"maas-controller-f664f4546-ck9lx\" (UID: \"8e60152e-fb27-4d18-991f-88975edfb782\") " pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:44:56.060170 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.060027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-metrics-certs\") pod \"maas-controller-64c89c545-phszt\" (UID: \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\") " pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 04 23:44:56.060170 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.060074    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46s68\" (UniqueName: \"kubernetes.io/projected/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-kube-api-access-46s68\") pod \"maas-controller-64c89c545-phszt\" (UID: \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\") " pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 04 23:44:56.060170 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:44:56.060091    2562 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 04 23:44:56.060170 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.060110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-metrics-certs\") pod \"maas-controller-f664f4546-ck9lx\" (UID: \"8e60152e-fb27-4d18-991f-88975edfb782\") " pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:44:56.060170 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.060134    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-webhook-cert\") pod \"maas-controller-64c89c545-phszt\" (UID: \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\") " pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 04 23:44:56.060170 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:44:56.060165    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-webhook-cert podName:8e60152e-fb27-4d18-991f-88975edfb782 nodeName:}" failed. No retries permitted until 2026-09-04 23:44:56.560146109 +0000 UTC m=+508.806813181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-webhook-cert") pod "maas-controller-f664f4546-ck9lx" (UID: "8e60152e-fb27-4d18-991f-88975edfb782") : secret "maas-controller-webhook-cert" not found
Sep 04 23:44:56.060466 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:44:56.060228    2562 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 04 23:44:56.060466 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.060269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64v8b\" (UniqueName: \"kubernetes.io/projected/8e60152e-fb27-4d18-991f-88975edfb782-kube-api-access-64v8b\") pod \"maas-controller-f664f4546-ck9lx\" (UID: \"8e60152e-fb27-4d18-991f-88975edfb782\") " pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:44:56.060466 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:44:56.060277    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-metrics-certs podName:8e60152e-fb27-4d18-991f-88975edfb782 nodeName:}" failed. No retries permitted until 2026-09-04 23:44:56.560259457 +0000 UTC m=+508.806926519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-metrics-certs") pod "maas-controller-f664f4546-ck9lx" (UID: "8e60152e-fb27-4d18-991f-88975edfb782") : secret "maas-controller-metrics-tls" not found
Sep 04 23:44:56.068256 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.068230    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64v8b\" (UniqueName: \"kubernetes.io/projected/8e60152e-fb27-4d18-991f-88975edfb782-kube-api-access-64v8b\") pod \"maas-controller-f664f4546-ck9lx\" (UID: \"8e60152e-fb27-4d18-991f-88975edfb782\") " pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:44:56.161058 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.160968    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-metrics-certs\") pod \"maas-controller-64c89c545-phszt\" (UID: \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\") " pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 04 23:44:56.161058 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.161013    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46s68\" (UniqueName: \"kubernetes.io/projected/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-kube-api-access-46s68\") pod \"maas-controller-64c89c545-phszt\" (UID: \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\") " pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 04 23:44:56.161058 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.161036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-webhook-cert\") pod \"maas-controller-64c89c545-phszt\" (UID: \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\") " pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 04 23:44:56.161292 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:44:56.161137    2562 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 04 23:44:56.161292 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:44:56.161209    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-metrics-certs podName:12bcdb7b-565e-4bc6-844e-cfb7fcac0d29 nodeName:}" failed. No retries permitted until 2026-09-04 23:44:56.661189808 +0000 UTC m=+508.907856870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-metrics-certs") pod "maas-controller-64c89c545-phszt" (UID: "12bcdb7b-565e-4bc6-844e-cfb7fcac0d29") : secret "maas-controller-metrics-tls" not found
Sep 04 23:44:56.163526 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.163498    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-webhook-cert\") pod \"maas-controller-64c89c545-phszt\" (UID: \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\") " pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 04 23:44:56.169293 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.169270    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46s68\" (UniqueName: \"kubernetes.io/projected/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-kube-api-access-46s68\") pod \"maas-controller-64c89c545-phszt\" (UID: \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\") " pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 04 23:44:56.565079 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.565040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-metrics-certs\") pod \"maas-controller-f664f4546-ck9lx\" (UID: \"8e60152e-fb27-4d18-991f-88975edfb782\") " pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:44:56.565259 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.565100    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-webhook-cert\") pod \"maas-controller-f664f4546-ck9lx\" (UID: \"8e60152e-fb27-4d18-991f-88975edfb782\") " pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:44:56.567530 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.567498    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-webhook-cert\") pod \"maas-controller-f664f4546-ck9lx\" (UID: \"8e60152e-fb27-4d18-991f-88975edfb782\") " pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:44:56.567639 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.567619    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-metrics-certs\") pod \"maas-controller-f664f4546-ck9lx\" (UID: \"8e60152e-fb27-4d18-991f-88975edfb782\") " pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:44:56.665461 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.665422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-metrics-certs\") pod \"maas-controller-64c89c545-phszt\" (UID: \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\") " pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 04 23:44:56.667900 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.667874    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-metrics-certs\") pod \"maas-controller-64c89c545-phszt\" (UID: \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\") " pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 04 23:44:56.736819 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.736782    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:44:56.899014 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:56.898929    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 04 23:44:57.006830 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:57.006797    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f664f4546-ck9lx"]
Sep 04 23:44:57.008496 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:44:57.008469    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e60152e_fb27_4d18_991f_88975edfb782.slice/crio-018231f8b01eef8bbbb0fe08ea8dd007c15f63abe9cadbf1836fcb0ba136180d WatchSource:0}: Error finding container 018231f8b01eef8bbbb0fe08ea8dd007c15f63abe9cadbf1836fcb0ba136180d: Status 404 returned error can't find the container with id 018231f8b01eef8bbbb0fe08ea8dd007c15f63abe9cadbf1836fcb0ba136180d
Sep 04 23:44:57.165786 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:57.165759    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64c89c545-phszt"]
Sep 04 23:44:57.167721 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:44:57.167691    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bcdb7b_565e_4bc6_844e_cfb7fcac0d29.slice/crio-4127cff23bf1763f383fb774439aaea04a4cfbf2e095739303b84b6a712064b8 WatchSource:0}: Error finding container 4127cff23bf1763f383fb774439aaea04a4cfbf2e095739303b84b6a712064b8: Status 404 returned error can't find the container with id 4127cff23bf1763f383fb774439aaea04a4cfbf2e095739303b84b6a712064b8
Sep 04 23:44:57.856663 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:57.856598    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64c89c545-phszt" event={"ID":"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29","Type":"ContainerStarted","Data":"4127cff23bf1763f383fb774439aaea04a4cfbf2e095739303b84b6a712064b8"}
Sep 04 23:44:57.858583 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:57.858515    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f664f4546-ck9lx" event={"ID":"8e60152e-fb27-4d18-991f-88975edfb782","Type":"ContainerStarted","Data":"018231f8b01eef8bbbb0fe08ea8dd007c15f63abe9cadbf1836fcb0ba136180d"}
Sep 04 23:44:59.871828 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:59.871720    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64c89c545-phszt" event={"ID":"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29","Type":"ContainerStarted","Data":"55cf14b47c1d5f719b16027ab7350482e898139146fb6f679855d23781bea22c"}
Sep 04 23:44:59.872245 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:59.871831    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 04 23:44:59.873296 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:59.873269    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f664f4546-ck9lx" event={"ID":"8e60152e-fb27-4d18-991f-88975edfb782","Type":"ContainerStarted","Data":"349843e59eac6346b73dc40c9e7e86d4cdf268730d2b20d0aedfc786afe2dad3"}
Sep 04 23:44:59.873416 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:59.873393    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:44:59.894349 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:59.894308    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-64c89c545-phszt" podStartSLOduration=2.441372317 podStartE2EDuration="4.89429574s" totalImagesPullingTime="2.452923423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:44:59.893313132 +0000 UTC m=+512.139980212" watchObservedRunningTime="2026-09-04 23:44:59.89429574 +0000 UTC m=+512.140962812"
Sep 04 23:44:59.911543 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:44:59.911500    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f664f4546-ck9lx" podStartSLOduration=2.308455297 podStartE2EDuration="4.911486836s" totalImagesPullingTime="2.603031539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:44:59.909242563 +0000 UTC m=+512.155909645" watchObservedRunningTime="2026-09-04 23:44:59.911486836 +0000 UTC m=+512.158153915"
Sep 04 23:45:10.879775 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:10.879742    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:45:10.880227 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:10.879814    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 04 23:45:10.941011 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:10.940977    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f664f4546-ck9lx"]
Sep 04 23:45:10.941226 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:10.941178    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f664f4546-ck9lx" podUID="8e60152e-fb27-4d18-991f-88975edfb782" containerName="manager" containerID="cri-o://349843e59eac6346b73dc40c9e7e86d4cdf268730d2b20d0aedfc786afe2dad3" gracePeriod=30
Sep 04 23:45:11.186274 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.186243    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:45:11.294855 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.294825    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64v8b\" (UniqueName: \"kubernetes.io/projected/8e60152e-fb27-4d18-991f-88975edfb782-kube-api-access-64v8b\") pod \"8e60152e-fb27-4d18-991f-88975edfb782\" (UID: \"8e60152e-fb27-4d18-991f-88975edfb782\") "
Sep 04 23:45:11.295062 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.294873    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-webhook-cert\") pod \"8e60152e-fb27-4d18-991f-88975edfb782\" (UID: \"8e60152e-fb27-4d18-991f-88975edfb782\") "
Sep 04 23:45:11.295062 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.294927    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-metrics-certs\") pod \"8e60152e-fb27-4d18-991f-88975edfb782\" (UID: \"8e60152e-fb27-4d18-991f-88975edfb782\") "
Sep 04 23:45:11.297079 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.297051    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e60152e-fb27-4d18-991f-88975edfb782-kube-api-access-64v8b" (OuterVolumeSpecName: "kube-api-access-64v8b") pod "8e60152e-fb27-4d18-991f-88975edfb782" (UID: "8e60152e-fb27-4d18-991f-88975edfb782"). InnerVolumeSpecName "kube-api-access-64v8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:45:11.297183 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.297146    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8e60152e-fb27-4d18-991f-88975edfb782" (UID: "8e60152e-fb27-4d18-991f-88975edfb782"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:45:11.297232 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.297215    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8e60152e-fb27-4d18-991f-88975edfb782" (UID: "8e60152e-fb27-4d18-991f-88975edfb782"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:45:11.396301 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.396267    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64v8b\" (UniqueName: \"kubernetes.io/projected/8e60152e-fb27-4d18-991f-88975edfb782-kube-api-access-64v8b\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:45:11.396301 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.396296    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-webhook-cert\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:45:11.396301 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.396305    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e60152e-fb27-4d18-991f-88975edfb782-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:45:11.933314 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.933281    2562 generic.go:356] "Generic (PLEG): container finished" podID="8e60152e-fb27-4d18-991f-88975edfb782" containerID="349843e59eac6346b73dc40c9e7e86d4cdf268730d2b20d0aedfc786afe2dad3" exitCode=0
Sep 04 23:45:11.933714 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.933323    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f664f4546-ck9lx"
Sep 04 23:45:11.933714 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.933331    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f664f4546-ck9lx" event={"ID":"8e60152e-fb27-4d18-991f-88975edfb782","Type":"ContainerDied","Data":"349843e59eac6346b73dc40c9e7e86d4cdf268730d2b20d0aedfc786afe2dad3"}
Sep 04 23:45:11.933714 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.933373    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f664f4546-ck9lx" event={"ID":"8e60152e-fb27-4d18-991f-88975edfb782","Type":"ContainerDied","Data":"018231f8b01eef8bbbb0fe08ea8dd007c15f63abe9cadbf1836fcb0ba136180d"}
Sep 04 23:45:11.933714 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.933393    2562 scope.go:117] "RemoveContainer" containerID="349843e59eac6346b73dc40c9e7e86d4cdf268730d2b20d0aedfc786afe2dad3"
Sep 04 23:45:11.942477 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.942337    2562 scope.go:117] "RemoveContainer" containerID="349843e59eac6346b73dc40c9e7e86d4cdf268730d2b20d0aedfc786afe2dad3"
Sep 04 23:45:11.942617 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:45:11.942598    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"349843e59eac6346b73dc40c9e7e86d4cdf268730d2b20d0aedfc786afe2dad3\": container with ID starting with 349843e59eac6346b73dc40c9e7e86d4cdf268730d2b20d0aedfc786afe2dad3 not found: ID does not exist" containerID="349843e59eac6346b73dc40c9e7e86d4cdf268730d2b20d0aedfc786afe2dad3"
Sep 04 23:45:11.942668 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.942625    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"349843e59eac6346b73dc40c9e7e86d4cdf268730d2b20d0aedfc786afe2dad3"} err="failed to get container status \"349843e59eac6346b73dc40c9e7e86d4cdf268730d2b20d0aedfc786afe2dad3\": rpc error: code = NotFound desc = could not find container \"349843e59eac6346b73dc40c9e7e86d4cdf268730d2b20d0aedfc786afe2dad3\": container with ID starting with 349843e59eac6346b73dc40c9e7e86d4cdf268730d2b20d0aedfc786afe2dad3 not found: ID does not exist"
Sep 04 23:45:11.953200 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.953178    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f664f4546-ck9lx"]
Sep 04 23:45:11.955026 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:11.955005    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f664f4546-ck9lx"]
Sep 04 23:45:12.357825 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:12.357791    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e60152e-fb27-4d18-991f-88975edfb782" path="/var/lib/kubelet/pods/8e60152e-fb27-4d18-991f-88975edfb782/volumes"
Sep 04 23:45:33.869130 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.869088    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"]
Sep 04 23:45:33.869743 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.869687    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e60152e-fb27-4d18-991f-88975edfb782" containerName="manager"
Sep 04 23:45:33.869743 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.869709    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e60152e-fb27-4d18-991f-88975edfb782" containerName="manager"
Sep 04 23:45:33.869867 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.869826    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e60152e-fb27-4d18-991f-88975edfb782" containerName="manager"
Sep 04 23:45:33.874771 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.874749    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:33.877229 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.877205    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 04 23:45:33.877444 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.877296    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 04 23:45:33.886137 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.886114    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"]
Sep 04 23:45:33.898281 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.898250    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/499191ea-f914-491c-9c86-786bac33e8cc-openshift-service-ca\") pod \"maas-api-66b67c778f-lvtdr\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") " pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:33.898408 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.898304    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjzlb\" (UniqueName: \"kubernetes.io/projected/499191ea-f914-491c-9c86-786bac33e8cc-kube-api-access-sjzlb\") pod \"maas-api-66b67c778f-lvtdr\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") " pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:33.898408 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.898357    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/499191ea-f914-491c-9c86-786bac33e8cc-maas-api-tls\") pod \"maas-api-66b67c778f-lvtdr\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") " pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:33.898516 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.898413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/499191ea-f914-491c-9c86-786bac33e8cc-metrics-certs\") pod \"maas-api-66b67c778f-lvtdr\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") " pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:33.999221 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.999180    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjzlb\" (UniqueName: \"kubernetes.io/projected/499191ea-f914-491c-9c86-786bac33e8cc-kube-api-access-sjzlb\") pod \"maas-api-66b67c778f-lvtdr\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") " pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:33.999406 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.999243    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/499191ea-f914-491c-9c86-786bac33e8cc-maas-api-tls\") pod \"maas-api-66b67c778f-lvtdr\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") " pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:33.999406 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.999292    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/499191ea-f914-491c-9c86-786bac33e8cc-metrics-certs\") pod \"maas-api-66b67c778f-lvtdr\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") " pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:33.999406 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:33.999364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/499191ea-f914-491c-9c86-786bac33e8cc-openshift-service-ca\") pod \"maas-api-66b67c778f-lvtdr\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") " pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:33.999585 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:45:33.999410    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 04 23:45:33.999585 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:45:33.999478    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/499191ea-f914-491c-9c86-786bac33e8cc-maas-api-tls podName:499191ea-f914-491c-9c86-786bac33e8cc nodeName:}" failed. No retries permitted until 2026-09-04 23:45:34.499458737 +0000 UTC m=+546.746125798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/499191ea-f914-491c-9c86-786bac33e8cc-maas-api-tls") pod "maas-api-66b67c778f-lvtdr" (UID: "499191ea-f914-491c-9c86-786bac33e8cc") : secret "maas-api-serving-cert" not found
Sep 04 23:45:34.000162 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.000131    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/499191ea-f914-491c-9c86-786bac33e8cc-openshift-service-ca\") pod \"maas-api-66b67c778f-lvtdr\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") " pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:34.002129 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.002108    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/499191ea-f914-491c-9c86-786bac33e8cc-metrics-certs\") pod \"maas-api-66b67c778f-lvtdr\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") " pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:34.009812 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.009777    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjzlb\" (UniqueName: \"kubernetes.io/projected/499191ea-f914-491c-9c86-786bac33e8cc-kube-api-access-sjzlb\") pod \"maas-api-66b67c778f-lvtdr\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") " pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:34.504199 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.504168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/499191ea-f914-491c-9c86-786bac33e8cc-maas-api-tls\") pod \"maas-api-66b67c778f-lvtdr\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") " pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:34.506603 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.506579    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/499191ea-f914-491c-9c86-786bac33e8cc-maas-api-tls\") pod \"maas-api-66b67c778f-lvtdr\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") " pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:34.511284 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.511260    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7nvhm"]
Sep 04 23:45:34.513843 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.513826    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7nvhm"
Sep 04 23:45:34.516099 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.516082    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 04 23:45:34.516175 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.516117    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-j6v2z\""
Sep 04 23:45:34.525578 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.525555    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7nvhm"]
Sep 04 23:45:34.604645 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.604607    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b9fe449-ea11-4359-bf6e-e430169a274b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7nvhm\" (UID: \"0b9fe449-ea11-4359-bf6e-e430169a274b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7nvhm"
Sep 04 23:45:34.604807 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.604734    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cbw5\" (UniqueName: \"kubernetes.io/projected/0b9fe449-ea11-4359-bf6e-e430169a274b-kube-api-access-7cbw5\") pod \"payload-processing-5b54b7fbb7-7nvhm\" (UID: \"0b9fe449-ea11-4359-bf6e-e430169a274b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7nvhm"
Sep 04 23:45:34.705297 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.705255    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cbw5\" (UniqueName: \"kubernetes.io/projected/0b9fe449-ea11-4359-bf6e-e430169a274b-kube-api-access-7cbw5\") pod \"payload-processing-5b54b7fbb7-7nvhm\" (UID: \"0b9fe449-ea11-4359-bf6e-e430169a274b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7nvhm"
Sep 04 23:45:34.705462 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.705318    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b9fe449-ea11-4359-bf6e-e430169a274b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7nvhm\" (UID: \"0b9fe449-ea11-4359-bf6e-e430169a274b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7nvhm"
Sep 04 23:45:34.705995 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.705976    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b9fe449-ea11-4359-bf6e-e430169a274b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7nvhm\" (UID: \"0b9fe449-ea11-4359-bf6e-e430169a274b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7nvhm"
Sep 04 23:45:34.713239 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.713216    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cbw5\" (UniqueName: \"kubernetes.io/projected/0b9fe449-ea11-4359-bf6e-e430169a274b-kube-api-access-7cbw5\") pod \"payload-processing-5b54b7fbb7-7nvhm\" (UID: \"0b9fe449-ea11-4359-bf6e-e430169a274b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7nvhm"
Sep 04 23:45:34.786339 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.786254    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:34.825346 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:34.825314    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7nvhm"
Sep 04 23:45:35.055627 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:35.055555    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"]
Sep 04 23:45:35.059089 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:45:35.059056    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499191ea_f914_491c_9c86_786bac33e8cc.slice/crio-ee267e549a0c46da55dd1ca5bbdeaa37684774208a24ee8cdb8c48f645dbce5c WatchSource:0}: Error finding container ee267e549a0c46da55dd1ca5bbdeaa37684774208a24ee8cdb8c48f645dbce5c: Status 404 returned error can't find the container with id ee267e549a0c46da55dd1ca5bbdeaa37684774208a24ee8cdb8c48f645dbce5c
Sep 04 23:45:35.110114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:35.110086    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7nvhm"]
Sep 04 23:45:35.112705 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:45:35.112676    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9fe449_ea11_4359_bf6e_e430169a274b.slice/crio-23d7454cc78469bfd0a5d87b34d04dc0dc381097aee8adc3593dab2026121562 WatchSource:0}: Error finding container 23d7454cc78469bfd0a5d87b34d04dc0dc381097aee8adc3593dab2026121562: Status 404 returned error can't find the container with id 23d7454cc78469bfd0a5d87b34d04dc0dc381097aee8adc3593dab2026121562
Sep 04 23:45:36.056788 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:36.056736    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr" event={"ID":"499191ea-f914-491c-9c86-786bac33e8cc","Type":"ContainerStarted","Data":"ee267e549a0c46da55dd1ca5bbdeaa37684774208a24ee8cdb8c48f645dbce5c"}
Sep 04 23:45:36.058638 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:36.058574    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7nvhm" event={"ID":"0b9fe449-ea11-4359-bf6e-e430169a274b","Type":"ContainerStarted","Data":"23d7454cc78469bfd0a5d87b34d04dc0dc381097aee8adc3593dab2026121562"}
Sep 04 23:45:39.080607 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:39.080253    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr" event={"ID":"499191ea-f914-491c-9c86-786bac33e8cc","Type":"ContainerStarted","Data":"7125c0f8d8ff436d3bb5852fce011a28f7fa25f1ac4cffe90c237703e24a0e16"}
Sep 04 23:45:39.080607 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:39.080453    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:39.081936 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:39.081910    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7nvhm" event={"ID":"0b9fe449-ea11-4359-bf6e-e430169a274b","Type":"ContainerStarted","Data":"af810005e74064da1561f77898cf3517db279ead30774fd94c6b37991a87c7e2"}
Sep 04 23:45:39.082049 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:39.081978    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7nvhm"
Sep 04 23:45:39.096043 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:39.095994    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr" podStartSLOduration=2.4143513309999998 podStartE2EDuration="6.095978669s" totalImagesPullingTime="3.681627338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:45:39.094513317 +0000 UTC m=+551.341180393" watchObservedRunningTime="2026-09-04 23:45:39.095978669 +0000 UTC m=+551.342645749"
Sep 04 23:45:39.109638 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:39.109590    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-7nvhm" podStartSLOduration=1.435202351 podStartE2EDuration="5.109574812s" totalImagesPullingTime="3.674372461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:45:39.108157574 +0000 UTC m=+551.354824656" watchObservedRunningTime="2026-09-04 23:45:39.109574812 +0000 UTC m=+551.356241891"
Sep 04 23:45:44.107129 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:44.107040    2562 generic.go:356] "Generic (PLEG): container finished" podID="d9907f4b-6c2c-4b46-b927-ff56ad289e45" containerID="fdda3df137eacf2dc9df8ab37375225f0d227171e507fc14e832d544b2abe6b5" exitCode=0
Sep 04 23:45:44.107129 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:44.107095    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4" event={"ID":"d9907f4b-6c2c-4b46-b927-ff56ad289e45","Type":"ContainerDied","Data":"fdda3df137eacf2dc9df8ab37375225f0d227171e507fc14e832d544b2abe6b5"}
Sep 04 23:45:44.107726 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:44.107603    2562 scope.go:117] "RemoveContainer" containerID="fdda3df137eacf2dc9df8ab37375225f0d227171e507fc14e832d544b2abe6b5"
Sep 04 23:45:44.108636 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:44.108605    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:45:44.108719 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:44.108694    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:45:44.108757 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:44.108733    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:45:45.091578 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:45.091546    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 04 23:45:45.114871 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:45.114835    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4" event={"ID":"d9907f4b-6c2c-4b46-b927-ff56ad289e45","Type":"ContainerStarted","Data":"4c28ff60c9e0763470dc2688b066f355f53a5ec8b86992512b6f4a04b38f823f"}
Sep 04 23:45:45.639288 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:45.639245    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:45:45.639503 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:45.639483    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-ln8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 04 23:45:45.639547 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:45.639527    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4" podUID="d9907f4b-6c2c-4b46-b927-ff56ad289e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 04 23:45:46.639734 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:46.639695    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-ln8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 04 23:45:46.640123 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:46.639757    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4" podUID="d9907f4b-6c2c-4b46-b927-ff56ad289e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 04 23:45:47.639380 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:47.639342    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-ln8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 04 23:45:47.639556 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:47.639433    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4" podUID="d9907f4b-6c2c-4b46-b927-ff56ad289e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 04 23:45:48.639746 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:48.639709    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-ln8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 04 23:45:48.640129 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:48.639773    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4" podUID="d9907f4b-6c2c-4b46-b927-ff56ad289e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 04 23:45:49.639983 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:49.639940    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-ln8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 04 23:45:49.640373 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:49.640008    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4" podUID="d9907f4b-6c2c-4b46-b927-ff56ad289e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 04 23:45:50.089241 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:50.089204    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7nvhm"
Sep 04 23:45:50.643375 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:50.643343    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:45:50.643742 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:50.643613    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:45:50.644266 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:50.644251    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ln8g4"
Sep 04 23:45:54.128950 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.128918    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"]
Sep 04 23:45:54.133084 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.133067    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.136489 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.136456    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rfvrs\""
Sep 04 23:45:54.136636 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.136509    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 04 23:45:54.136636 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.136511    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 04 23:45:54.136636 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.136456    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 04 23:45:54.140803 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.140780    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"]
Sep 04 23:45:54.212575 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.212544    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx24w\" (UniqueName: \"kubernetes.io/projected/35f164f5-e979-4dbc-b894-ff0681e0e758-kube-api-access-jx24w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.212785 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.212596    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f164f5-e979-4dbc-b894-ff0681e0e758-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.212785 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.212642    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f164f5-e979-4dbc-b894-ff0681e0e758-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.212785 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.212679    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f164f5-e979-4dbc-b894-ff0681e0e758-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.212785 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.212717    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f164f5-e979-4dbc-b894-ff0681e0e758-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.212959 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.212790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f164f5-e979-4dbc-b894-ff0681e0e758-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.313189 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.313155    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f164f5-e979-4dbc-b894-ff0681e0e758-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.313189 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.313196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f164f5-e979-4dbc-b894-ff0681e0e758-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.313441 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.313220    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f164f5-e979-4dbc-b894-ff0681e0e758-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.313441 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.313246    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f164f5-e979-4dbc-b894-ff0681e0e758-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.313441 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.313315    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx24w\" (UniqueName: \"kubernetes.io/projected/35f164f5-e979-4dbc-b894-ff0681e0e758-kube-api-access-jx24w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.313441 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.313367    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f164f5-e979-4dbc-b894-ff0681e0e758-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.313731 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.313704    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f164f5-e979-4dbc-b894-ff0681e0e758-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.313849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.313811    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f164f5-e979-4dbc-b894-ff0681e0e758-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.313933 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.313907    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f164f5-e979-4dbc-b894-ff0681e0e758-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.315470 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.315449    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f164f5-e979-4dbc-b894-ff0681e0e758-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.315755 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.315735    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f164f5-e979-4dbc-b894-ff0681e0e758-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.321017 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.320995    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx24w\" (UniqueName: \"kubernetes.io/projected/35f164f5-e979-4dbc-b894-ff0681e0e758-kube-api-access-jx24w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694\" (UID: \"35f164f5-e979-4dbc-b894-ff0681e0e758\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.446986 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.446895    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:45:54.720558 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:54.720532    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"]
Sep 04 23:45:54.722729 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:45:54.722707    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f164f5_e979_4dbc_b894_ff0681e0e758.slice/crio-8f848ee761ce66e31323f79275e18da2e1d0a40bd8ea3039c6fbdc3d545e21db WatchSource:0}: Error finding container 8f848ee761ce66e31323f79275e18da2e1d0a40bd8ea3039c6fbdc3d545e21db: Status 404 returned error can't find the container with id 8f848ee761ce66e31323f79275e18da2e1d0a40bd8ea3039c6fbdc3d545e21db
Sep 04 23:45:55.174047 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:55.173966    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694" event={"ID":"35f164f5-e979-4dbc-b894-ff0681e0e758","Type":"ContainerStarted","Data":"8f848ee761ce66e31323f79275e18da2e1d0a40bd8ea3039c6fbdc3d545e21db"}
Sep 04 23:45:56.180292 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:56.180192    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694" event={"ID":"35f164f5-e979-4dbc-b894-ff0681e0e758","Type":"ContainerStarted","Data":"7fb1da7e1edb0ab7ca95cb42e9537e21cfea8c0f1143f1997cfadc9f5a9383a0"}
Sep 04 23:45:56.198233 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:45:56.198179    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694" podStartSLOduration=1.104250251 podStartE2EDuration="2.198162925s" totalImagesPullingTime="1.093912674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:45:56.196248557 +0000 UTC m=+568.442915636" watchObservedRunningTime="2026-09-04 23:45:56.198162925 +0000 UTC m=+568.444830004"
Sep 04 23:46:04.447879 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:04.447842    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:46:04.448322 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:04.447910    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:46:04.460582 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:04.460558    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:46:05.243140 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:05.243115    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694"
Sep 04 23:46:14.822329 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.822293    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"]
Sep 04 23:46:14.827554 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.827532    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.829853 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.829833    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 04 23:46:14.834195 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.834163    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"]
Sep 04 23:46:14.894181 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.894145    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a4a6b81-c17e-45e9-b262-2a1b8da54710-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.894181 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.894183    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a4a6b81-c17e-45e9-b262-2a1b8da54710-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.894376 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.894209    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a4a6b81-c17e-45e9-b262-2a1b8da54710-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.894376 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.894296    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzj9v\" (UniqueName: \"kubernetes.io/projected/7a4a6b81-c17e-45e9-b262-2a1b8da54710-kube-api-access-wzj9v\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.894376 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.894332    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a4a6b81-c17e-45e9-b262-2a1b8da54710-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.894473 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.894389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a4a6b81-c17e-45e9-b262-2a1b8da54710-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.995725 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.995685    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a4a6b81-c17e-45e9-b262-2a1b8da54710-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.996013 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.995736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a4a6b81-c17e-45e9-b262-2a1b8da54710-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.996013 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.995842    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a4a6b81-c17e-45e9-b262-2a1b8da54710-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.996013 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.995897    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzj9v\" (UniqueName: \"kubernetes.io/projected/7a4a6b81-c17e-45e9-b262-2a1b8da54710-kube-api-access-wzj9v\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.996013 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.995927    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a4a6b81-c17e-45e9-b262-2a1b8da54710-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.996013 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.995961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a4a6b81-c17e-45e9-b262-2a1b8da54710-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.996301 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.996146    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a4a6b81-c17e-45e9-b262-2a1b8da54710-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.996428 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.996403    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a4a6b81-c17e-45e9-b262-2a1b8da54710-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.996510 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.996485    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a4a6b81-c17e-45e9-b262-2a1b8da54710-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.998707 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.998683    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a4a6b81-c17e-45e9-b262-2a1b8da54710-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:14.999091 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:14.999067    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a4a6b81-c17e-45e9-b262-2a1b8da54710-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:15.003394 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:15.003368    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzj9v\" (UniqueName: \"kubernetes.io/projected/7a4a6b81-c17e-45e9-b262-2a1b8da54710-kube-api-access-wzj9v\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf\" (UID: \"7a4a6b81-c17e-45e9-b262-2a1b8da54710\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:15.139560 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:15.139460    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:15.409541 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:15.409518    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"]
Sep 04 23:46:15.411297 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:46:15.411270    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a4a6b81_c17e_45e9_b262_2a1b8da54710.slice/crio-6c3990657c9239bbd6b70c9f625caec3ac315f6eb055a107a744993543c53f9e WatchSource:0}: Error finding container 6c3990657c9239bbd6b70c9f625caec3ac315f6eb055a107a744993543c53f9e: Status 404 returned error can't find the container with id 6c3990657c9239bbd6b70c9f625caec3ac315f6eb055a107a744993543c53f9e
Sep 04 23:46:16.286480 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:16.286439    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf" event={"ID":"7a4a6b81-c17e-45e9-b262-2a1b8da54710","Type":"ContainerStarted","Data":"077d1b8b45d1440d37f6bc8aa2548c522aa6ed8e9d2f0bb3aadb90cd0dc7239b"}
Sep 04 23:46:16.287015 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:16.286496    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf" event={"ID":"7a4a6b81-c17e-45e9-b262-2a1b8da54710","Type":"ContainerStarted","Data":"6c3990657c9239bbd6b70c9f625caec3ac315f6eb055a107a744993543c53f9e"}
Sep 04 23:46:16.332399 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:16.332343    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf" podStartSLOduration=2.159202565 podStartE2EDuration="2.332326797s" totalImagesPullingTime="173.124232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:46:16.329561829 +0000 UTC m=+588.576228907" watchObservedRunningTime="2026-09-04 23:46:16.332326797 +0000 UTC m=+588.578993876"
Sep 04 23:46:25.140060 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:25.140010    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:25.140690 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:25.140084    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:25.152835 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:25.152807    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:25.342417 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:25.342381    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf"
Sep 04 23:46:28.204407 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:28.204377    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 04 23:46:28.205848 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:28.205823    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 04 23:46:28.208466 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:28.208442    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 04 23:46:28.209624 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:28.209604    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 04 23:46:43.620587 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.620549    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"]
Sep 04 23:46:43.626269 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.626246    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.628629 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.628607    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 04 23:46:43.635016 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.634707    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"]
Sep 04 23:46:43.752731 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.752694    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.752932 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.752757    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.752932 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.752858    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.752932 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.752903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42dxj\" (UniqueName: \"kubernetes.io/projected/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-kube-api-access-42dxj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.753074 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.752932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.753074 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.753055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.854173 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.854123    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42dxj\" (UniqueName: \"kubernetes.io/projected/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-kube-api-access-42dxj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.854372 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.854192    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.854372 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.854237    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.854372 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.854271    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.854540 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.854475    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.854689 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.854570    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.854760 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.854740    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.854834 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.854801    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.855120 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.855091    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.857151 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.857120    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.857281 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.857183    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.862268 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.862234    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42dxj\" (UniqueName: \"kubernetes.io/projected/f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee-kube-api-access-42dxj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qc2fl\" (UID: \"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:43.939066 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:43.938969    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:44.218836 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:44.218800    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"]
Sep 04 23:46:44.220615 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:46:44.220588    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ddf861_9ca0_4d4d_a3e8_ace18e9f48ee.slice/crio-574748763a29a0fd518f45b21273b53469a4ecf73c1ef8a522990af9af3d592b WatchSource:0}: Error finding container 574748763a29a0fd518f45b21273b53469a4ecf73c1ef8a522990af9af3d592b: Status 404 returned error can't find the container with id 574748763a29a0fd518f45b21273b53469a4ecf73c1ef8a522990af9af3d592b
Sep 04 23:46:44.222018 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:44.222003    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 23:46:44.425846 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:44.425814    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl" event={"ID":"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee","Type":"ContainerStarted","Data":"574748763a29a0fd518f45b21273b53469a4ecf73c1ef8a522990af9af3d592b"}
Sep 04 23:46:45.432550 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:45.432510    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl" event={"ID":"f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee","Type":"ContainerStarted","Data":"0516a90affe5af7cce5e8feb651ef2a8be775ae5c828e0ae6b6d6587054eb281"}
Sep 04 23:46:45.451766 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:45.451715    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl" podStartSLOduration=2.272973208 podStartE2EDuration="2.45169967s" totalImagesPullingTime="178.726462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:46:45.449553155 +0000 UTC m=+617.696220237" watchObservedRunningTime="2026-09-04 23:46:45.45169967 +0000 UTC m=+617.698366758"
Sep 04 23:46:53.939390 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:53.939358    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:53.939806 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:53.939413    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:53.952145 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:53.952121    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:46:54.488434 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:46:54.488405    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qc2fl"
Sep 04 23:48:21.280629 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.280544    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"]
Sep 04 23:48:21.285006 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.284973    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:21.287321 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.287295    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d8b443-serving-cert\""
Sep 04 23:48:21.287438 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.287419    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mnd5p\""
Sep 04 23:48:21.292234 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.292209    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"]
Sep 04 23:48:21.330129 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.330085    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b631a9-360b-4eab-a373-d420d2fba2a9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:21.330315 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.330143    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vl7\" (UniqueName: \"kubernetes.io/projected/76b631a9-360b-4eab-a373-d420d2fba2a9-kube-api-access-k5vl7\") pod \"maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:21.330315 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.330196    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b631a9-360b-4eab-a373-d420d2fba2a9-metrics-certs\") pod \"maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:21.330315 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.330291    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b631a9-360b-4eab-a373-d420d2fba2a9-maas-api-tls\") pod \"maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:21.430884 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.430852    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b631a9-360b-4eab-a373-d420d2fba2a9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:21.431085 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.430900    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vl7\" (UniqueName: \"kubernetes.io/projected/76b631a9-360b-4eab-a373-d420d2fba2a9-kube-api-access-k5vl7\") pod \"maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:21.431085 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.430933    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b631a9-360b-4eab-a373-d420d2fba2a9-metrics-certs\") pod \"maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:21.431085 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.430984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b631a9-360b-4eab-a373-d420d2fba2a9-maas-api-tls\") pod \"maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:21.431261 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:48:21.431126    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-serving-cert: secret "maas-api-e2e-inf-a-d8b443-serving-cert" not found
Sep 04 23:48:21.431261 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:48:21.431196    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b631a9-360b-4eab-a373-d420d2fba2a9-maas-api-tls podName:76b631a9-360b-4eab-a373-d420d2fba2a9 nodeName:}" failed. No retries permitted until 2026-09-04 23:48:21.931175631 +0000 UTC m=+714.177842705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76b631a9-360b-4eab-a373-d420d2fba2a9-maas-api-tls") pod "maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb" (UID: "76b631a9-360b-4eab-a373-d420d2fba2a9") : secret "maas-api-e2e-inf-a-d8b443-serving-cert" not found
Sep 04 23:48:21.431754 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.431723    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b631a9-360b-4eab-a373-d420d2fba2a9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:21.433729 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.433705    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b631a9-360b-4eab-a373-d420d2fba2a9-metrics-certs\") pod \"maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:21.438849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.438827    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vl7\" (UniqueName: \"kubernetes.io/projected/76b631a9-360b-4eab-a373-d420d2fba2a9-kube-api-access-k5vl7\") pod \"maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:21.935845 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.935805    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b631a9-360b-4eab-a373-d420d2fba2a9-maas-api-tls\") pod \"maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:21.938448 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:21.938420    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b631a9-360b-4eab-a373-d420d2fba2a9-maas-api-tls\") pod \"maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:22.201416 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:22.201327    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:22.477724 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:22.477697    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"]
Sep 04 23:48:22.479503 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:48:22.479479    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b631a9_360b_4eab_a373_d420d2fba2a9.slice/crio-147aceb1b0a82ccb7efe74fccaeac6a4af51619550192c431b9f25691c5152b7 WatchSource:0}: Error finding container 147aceb1b0a82ccb7efe74fccaeac6a4af51619550192c431b9f25691c5152b7: Status 404 returned error can't find the container with id 147aceb1b0a82ccb7efe74fccaeac6a4af51619550192c431b9f25691c5152b7
Sep 04 23:48:22.912875 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:22.912791    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb" event={"ID":"76b631a9-360b-4eab-a373-d420d2fba2a9","Type":"ContainerStarted","Data":"147aceb1b0a82ccb7efe74fccaeac6a4af51619550192c431b9f25691c5152b7"}
Sep 04 23:48:23.919531 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:23.919495    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb" event={"ID":"76b631a9-360b-4eab-a373-d420d2fba2a9","Type":"ContainerStarted","Data":"be820bb5877e88b5a45606ece7b2c50a88afd686d69e56d1775b281db67d7dee"}
Sep 04 23:48:23.919965 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:23.919696    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:23.935816 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:23.935766    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb" podStartSLOduration=2.376162228 podStartE2EDuration="2.935749692s" totalImagesPullingTime="559.587464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:48:23.933862242 +0000 UTC m=+716.180529322" watchObservedRunningTime="2026-09-04 23:48:23.935749692 +0000 UTC m=+716.182416770"
Sep 04 23:48:25.661186 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.661153    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"]
Sep 04 23:48:25.665316 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.665295    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.667609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.667584    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-3c6688-openshift-default-dockercfg-vdngb\""
Sep 04 23:48:25.677139 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.677112    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"]
Sep 04 23:48:25.773239 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.773204    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-workload-certs\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.773239 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.773243    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-token\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.773504 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.773261    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-podinfo\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.773504 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.773293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-data\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.773504 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.773400    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-envoy\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.773504 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.773461    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-credential-socket\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.773803 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.773520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istiod-ca-cert\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.773803 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.773577    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-workload-socket\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.773803 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.773635    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lfnx\" (UniqueName: \"kubernetes.io/projected/976dc20d-a7b7-4894-bfd6-d2c84141c97e-kube-api-access-8lfnx\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.874883 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.874849    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-envoy\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.875083 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.874898    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-credential-socket\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.875083 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.874932    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istiod-ca-cert\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.875083 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.874961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-workload-socket\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.875083 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.874989    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lfnx\" (UniqueName: \"kubernetes.io/projected/976dc20d-a7b7-4894-bfd6-d2c84141c97e-kube-api-access-8lfnx\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.875083 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.875004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-workload-certs\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.875083 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.875023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-token\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.875083 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.875048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-podinfo\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.875449 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.875091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-data\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.875521 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.875499    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-credential-socket\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.875579 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.875534    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-workload-certs\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.875638 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.875574    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-workload-socket\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.875638 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.875591    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-data\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.876057 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.876035    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istiod-ca-cert\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.877304 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.877281    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-envoy\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.877740 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.877723    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-podinfo\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.882978 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.882952    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-token\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.883425 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.883409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lfnx\" (UniqueName: \"kubernetes.io/projected/976dc20d-a7b7-4894-bfd6-d2c84141c97e-kube-api-access-8lfnx\") pod \"e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") " pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:25.976843 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:25.976813    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:26.250005 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:26.249978    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"]
Sep 04 23:48:26.251540 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:48:26.251513    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976dc20d_a7b7_4894_bfd6_d2c84141c97e.slice/crio-810a9836545717503bc2e23d4cf578b519172d49fa1221cd731ffea809dec81b WatchSource:0}: Error finding container 810a9836545717503bc2e23d4cf578b519172d49fa1221cd731ffea809dec81b: Status 404 returned error can't find the container with id 810a9836545717503bc2e23d4cf578b519172d49fa1221cd731ffea809dec81b
Sep 04 23:48:26.253852 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:26.253823    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:48:26.253920 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:26.253888    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:48:26.253960 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:26.253918    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:48:26.936995 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:26.936960    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn" event={"ID":"976dc20d-a7b7-4894-bfd6-d2c84141c97e","Type":"ContainerStarted","Data":"8036f75e2036dcd7a8f307a84344d52028e313b360c351ea2c5f95f3aed1846b"}
Sep 04 23:48:26.937388 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:26.937009    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn" event={"ID":"976dc20d-a7b7-4894-bfd6-d2c84141c97e","Type":"ContainerStarted","Data":"810a9836545717503bc2e23d4cf578b519172d49fa1221cd731ffea809dec81b"}
Sep 04 23:48:26.956549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:26.956502    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn" podStartSLOduration=1.956488492 podStartE2EDuration="1.956488492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:48:26.955481549 +0000 UTC m=+719.202148631" watchObservedRunningTime="2026-09-04 23:48:26.956488492 +0000 UTC m=+719.203155570"
Sep 04 23:48:26.977922 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:26.977890    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:27.982506 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:27.982472    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:28.889675 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:28.889403    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"]
Sep 04 23:48:28.894472 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:28.894443    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:28.896865 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:28.896840    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-eef2d470-serving-cert\""
Sep 04 23:48:28.910277 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:28.910253    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"]
Sep 04 23:48:28.948263 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:28.948232    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:28.949368 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:28.949349    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:29.001087 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.001050    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9923ebd1-57fe-4100-87dd-2c293fb22fd4-metrics-certs\") pod \"maas-api-e2e-ait-eef2d470-57675df759-n42gz\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:29.001475 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.001283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9923ebd1-57fe-4100-87dd-2c293fb22fd4-maas-api-tls\") pod \"maas-api-e2e-ait-eef2d470-57675df759-n42gz\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:29.001475 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.001319    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64hrb\" (UniqueName: \"kubernetes.io/projected/9923ebd1-57fe-4100-87dd-2c293fb22fd4-kube-api-access-64hrb\") pod \"maas-api-e2e-ait-eef2d470-57675df759-n42gz\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:29.001475 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.001433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9923ebd1-57fe-4100-87dd-2c293fb22fd4-openshift-service-ca\") pod \"maas-api-e2e-ait-eef2d470-57675df759-n42gz\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:29.102439 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.102400    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9923ebd1-57fe-4100-87dd-2c293fb22fd4-maas-api-tls\") pod \"maas-api-e2e-ait-eef2d470-57675df759-n42gz\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:29.102439 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.102441    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64hrb\" (UniqueName: \"kubernetes.io/projected/9923ebd1-57fe-4100-87dd-2c293fb22fd4-kube-api-access-64hrb\") pod \"maas-api-e2e-ait-eef2d470-57675df759-n42gz\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:29.102743 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.102495    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9923ebd1-57fe-4100-87dd-2c293fb22fd4-openshift-service-ca\") pod \"maas-api-e2e-ait-eef2d470-57675df759-n42gz\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:29.102743 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.102545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9923ebd1-57fe-4100-87dd-2c293fb22fd4-metrics-certs\") pod \"maas-api-e2e-ait-eef2d470-57675df759-n42gz\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:29.103259 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.103237    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9923ebd1-57fe-4100-87dd-2c293fb22fd4-openshift-service-ca\") pod \"maas-api-e2e-ait-eef2d470-57675df759-n42gz\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:29.104935 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.104913    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9923ebd1-57fe-4100-87dd-2c293fb22fd4-metrics-certs\") pod \"maas-api-e2e-ait-eef2d470-57675df759-n42gz\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:29.104935 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.104929    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9923ebd1-57fe-4100-87dd-2c293fb22fd4-maas-api-tls\") pod \"maas-api-e2e-ait-eef2d470-57675df759-n42gz\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:29.116820 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.116796    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64hrb\" (UniqueName: \"kubernetes.io/projected/9923ebd1-57fe-4100-87dd-2c293fb22fd4-kube-api-access-64hrb\") pod \"maas-api-e2e-ait-eef2d470-57675df759-n42gz\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:29.208841 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.208789    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:29.481693 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.481662    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"]
Sep 04 23:48:29.484061 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:48:29.484030    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9923ebd1_57fe_4100_87dd_2c293fb22fd4.slice/crio-4a66d2038020c3f65232bf9c103f1d1b67422674bab85249027dfbd9a4ada8b3 WatchSource:0}: Error finding container 4a66d2038020c3f65232bf9c103f1d1b67422674bab85249027dfbd9a4ada8b3: Status 404 returned error can't find the container with id 4a66d2038020c3f65232bf9c103f1d1b67422674bab85249027dfbd9a4ada8b3
Sep 04 23:48:29.931111 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.931059    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:48:29.954578 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.954547    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz" event={"ID":"9923ebd1-57fe-4100-87dd-2c293fb22fd4","Type":"ContainerStarted","Data":"46dae7dae094e3d84f7ed24f447be6c2331a4a66a395593745074ddac7859065"}
Sep 04 23:48:29.954882 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.954595    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz" event={"ID":"9923ebd1-57fe-4100-87dd-2c293fb22fd4","Type":"ContainerStarted","Data":"4a66d2038020c3f65232bf9c103f1d1b67422674bab85249027dfbd9a4ada8b3"}
Sep 04 23:48:29.954882 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.954801    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:29.972467 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:29.972415    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz" podStartSLOduration=1.669618178 podStartE2EDuration="1.972398008s" totalImagesPullingTime="302.77983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:48:29.969899248 +0000 UTC m=+722.216566341" watchObservedRunningTime="2026-09-04 23:48:29.972398008 +0000 UTC m=+722.219065086"
Sep 04 23:48:30.464155 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:30.463891    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"]
Sep 04 23:48:30.467885 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:30.467867    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:30.469898 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:30.469871    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-3c6688-serving-cert\""
Sep 04 23:48:30.483336 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:30.483308    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"]
Sep 04 23:48:30.515252 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:30.515221    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npcg4\" (UniqueName: \"kubernetes.io/projected/0032a5be-72f4-465a-a5a9-3b09880ae938-kube-api-access-npcg4\") pod \"maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:30.515396 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:30.515323    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0032a5be-72f4-465a-a5a9-3b09880ae938-metrics-certs\") pod \"maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:30.515396 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:30.515366    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0032a5be-72f4-465a-a5a9-3b09880ae938-openshift-service-ca\") pod \"maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:30.515396 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:30.515384    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0032a5be-72f4-465a-a5a9-3b09880ae938-maas-api-tls\") pod \"maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:30.616683 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:30.616640    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0032a5be-72f4-465a-a5a9-3b09880ae938-metrics-certs\") pod \"maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:30.616856 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:30.616707    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0032a5be-72f4-465a-a5a9-3b09880ae938-openshift-service-ca\") pod \"maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:30.616856 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:30.616728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0032a5be-72f4-465a-a5a9-3b09880ae938-maas-api-tls\") pod \"maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:30.616856 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:30.616762    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npcg4\" (UniqueName: \"kubernetes.io/projected/0032a5be-72f4-465a-a5a9-3b09880ae938-kube-api-access-npcg4\") pod \"maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:30.617012 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:48:30.616877    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-serving-cert: secret "maas-api-e2e-autores-3c6688-serving-cert" not found
Sep 04 23:48:30.617012 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:48:30.616948    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0032a5be-72f4-465a-a5a9-3b09880ae938-maas-api-tls podName:0032a5be-72f4-465a-a5a9-3b09880ae938 nodeName:}" failed. No retries permitted until 2026-09-04 23:48:31.116927828 +0000 UTC m=+723.363594899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0032a5be-72f4-465a-a5a9-3b09880ae938-maas-api-tls") pod "maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl" (UID: "0032a5be-72f4-465a-a5a9-3b09880ae938") : secret "maas-api-e2e-autores-3c6688-serving-cert" not found
Sep 04 23:48:30.617455 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:30.617429    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0032a5be-72f4-465a-a5a9-3b09880ae938-openshift-service-ca\") pod \"maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:30.619162 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:30.619134    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0032a5be-72f4-465a-a5a9-3b09880ae938-metrics-certs\") pod \"maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:30.627492 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:30.627472    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npcg4\" (UniqueName: \"kubernetes.io/projected/0032a5be-72f4-465a-a5a9-3b09880ae938-kube-api-access-npcg4\") pod \"maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:31.122315 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:31.122278    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0032a5be-72f4-465a-a5a9-3b09880ae938-maas-api-tls\") pod \"maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:31.124740 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:31.124714    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0032a5be-72f4-465a-a5a9-3b09880ae938-maas-api-tls\") pod \"maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:31.384105 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:31.384020    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:31.657699 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:31.657647    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"]
Sep 04 23:48:31.658609 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:48:31.658584    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0032a5be_72f4_465a_a5a9_3b09880ae938.slice/crio-575846af831f7dc919486985ea972c22313937ab7705a6fffc8a273d023a0fe5 WatchSource:0}: Error finding container 575846af831f7dc919486985ea972c22313937ab7705a6fffc8a273d023a0fe5: Status 404 returned error can't find the container with id 575846af831f7dc919486985ea972c22313937ab7705a6fffc8a273d023a0fe5
Sep 04 23:48:31.966362 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:31.966331    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl" event={"ID":"0032a5be-72f4-465a-a5a9-3b09880ae938","Type":"ContainerStarted","Data":"575846af831f7dc919486985ea972c22313937ab7705a6fffc8a273d023a0fe5"}
Sep 04 23:48:32.973467 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:32.973433    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl" event={"ID":"0032a5be-72f4-465a-a5a9-3b09880ae938","Type":"ContainerStarted","Data":"374f83cdd3783146817a5282752b96659a251367b7a2deea0ecddb72be0eadb3"}
Sep 04 23:48:32.973887 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:32.973534    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:32.992957 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:32.992910    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl" podStartSLOduration=2.5937337940000003 podStartE2EDuration="2.992893878s" totalImagesPullingTime="399.160084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:48:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:48:32.991675263 +0000 UTC m=+725.238342344" watchObservedRunningTime="2026-09-04 23:48:32.992893878 +0000 UTC m=+725.239560957"
Sep 04 23:48:35.966284 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:35.966252    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:48:38.984251 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:38.984223    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:48:44.760308 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:44.760105    2562 patch_prober.go:28] interesting pod/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:47254->10.132.0.51:15021: read: connection reset by peer" start-of-body=
Sep 04 23:48:44.760308 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:44.760216    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn" podUID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:47254->10.132.0.51:15021: read: connection reset by peer"
Sep 04 23:48:45.039637 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:45.039556    2562 generic.go:356] "Generic (PLEG): container finished" podID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerID="8036f75e2036dcd7a8f307a84344d52028e313b360c351ea2c5f95f3aed1846b" exitCode=0
Sep 04 23:48:45.039801 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:45.039632    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn" event={"ID":"976dc20d-a7b7-4894-bfd6-d2c84141c97e","Type":"ContainerDied","Data":"8036f75e2036dcd7a8f307a84344d52028e313b360c351ea2c5f95f3aed1846b"}
Sep 04 23:48:45.040142 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:45.040124    2562 scope.go:117] "RemoveContainer" containerID="8036f75e2036dcd7a8f307a84344d52028e313b360c351ea2c5f95f3aed1846b"
Sep 04 23:48:45.041192 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:45.041165    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:48:45.041282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:45.041228    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:48:45.041282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:45.041260    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:48:46.048206 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:46.048171    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn" event={"ID":"976dc20d-a7b7-4894-bfd6-d2c84141c97e","Type":"ContainerStarted","Data":"d6686147663bcbd9494f5baad001e4c881497e22161e7b91f7d625f640cd216d"}
Sep 04 23:48:46.977444 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:46.977404    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:46.977631 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:46.977589    2562 patch_prober.go:28] interesting pod/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 04 23:48:46.977710 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:46.977638    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn" podUID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 04 23:48:47.977842 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:47.977801    2562 patch_prober.go:28] interesting pod/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 04 23:48:47.978229 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:47.977866    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn" podUID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 04 23:48:48.977251 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:48.977207    2562 patch_prober.go:28] interesting pod/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 04 23:48:48.977468 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:48.977273    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn" podUID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 04 23:48:49.557471 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.557433    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"]
Sep 04 23:48:49.560111 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.560094    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:49.562380 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.562358    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-724e1f-serving-cert\""
Sep 04 23:48:49.571785 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.571758    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"]
Sep 04 23:48:49.719645 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.719600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0677400f-dc86-4217-b126-b807feeabecd-metrics-certs\") pod \"maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:49.719942 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.719915    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0677400f-dc86-4217-b126-b807feeabecd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:49.720140 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.720121    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0677400f-dc86-4217-b126-b807feeabecd-maas-api-tls\") pod \"maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:49.720336 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.720318    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mdzb\" (UniqueName: \"kubernetes.io/projected/0677400f-dc86-4217-b126-b807feeabecd-kube-api-access-2mdzb\") pod \"maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:49.821845 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.821740    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mdzb\" (UniqueName: \"kubernetes.io/projected/0677400f-dc86-4217-b126-b807feeabecd-kube-api-access-2mdzb\") pod \"maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:49.822494 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.822469    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0677400f-dc86-4217-b126-b807feeabecd-metrics-certs\") pod \"maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:49.823136 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.823117    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0677400f-dc86-4217-b126-b807feeabecd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:49.823347 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.823330    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0677400f-dc86-4217-b126-b807feeabecd-maas-api-tls\") pod \"maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:49.823670 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:48:49.823638    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-serving-cert: secret "maas-api-e2e-inf-b-724e1f-serving-cert" not found
Sep 04 23:48:49.823847 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:48:49.823834    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0677400f-dc86-4217-b126-b807feeabecd-maas-api-tls podName:0677400f-dc86-4217-b126-b807feeabecd nodeName:}" failed. No retries permitted until 2026-09-04 23:48:50.323812667 +0000 UTC m=+742.570479726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0677400f-dc86-4217-b126-b807feeabecd-maas-api-tls") pod "maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn" (UID: "0677400f-dc86-4217-b126-b807feeabecd") : secret "maas-api-e2e-inf-b-724e1f-serving-cert" not found
Sep 04 23:48:49.823950 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.823877    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0677400f-dc86-4217-b126-b807feeabecd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:49.826748 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.826726    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0677400f-dc86-4217-b126-b807feeabecd-metrics-certs\") pod \"maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:49.838306 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.838275    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mdzb\" (UniqueName: \"kubernetes.io/projected/0677400f-dc86-4217-b126-b807feeabecd-kube-api-access-2mdzb\") pod \"maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:49.977681 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.977624    2562 patch_prober.go:28] interesting pod/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 04 23:48:49.977904 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:49.977714    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn" podUID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 04 23:48:50.337677 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:50.335041    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0677400f-dc86-4217-b126-b807feeabecd-maas-api-tls\") pod \"maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:50.341675 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:50.338849    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0677400f-dc86-4217-b126-b807feeabecd-maas-api-tls\") pod \"maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:50.473968 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:50.473929    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:50.764555 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:50.764524    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"]
Sep 04 23:48:50.765700 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:48:50.765673    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0677400f_dc86_4217_b126_b807feeabecd.slice/crio-784e05dad0dffa7a7f2b67deca85461ee97d51fed08f1a90b0980844850f23a6 WatchSource:0}: Error finding container 784e05dad0dffa7a7f2b67deca85461ee97d51fed08f1a90b0980844850f23a6: Status 404 returned error can't find the container with id 784e05dad0dffa7a7f2b67deca85461ee97d51fed08f1a90b0980844850f23a6
Sep 04 23:48:50.977883 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:50.977847    2562 patch_prober.go:28] interesting pod/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 04 23:48:50.978050 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:50.977922    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn" podUID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 04 23:48:51.075166 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:51.075072    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn" event={"ID":"0677400f-dc86-4217-b126-b807feeabecd","Type":"ContainerStarted","Data":"784e05dad0dffa7a7f2b67deca85461ee97d51fed08f1a90b0980844850f23a6"}
Sep 04 23:48:51.982004 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:51.981973    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:51.982483 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:51.982355    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:51.983162 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:51.983146    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:48:52.082594 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:52.082547    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn" event={"ID":"0677400f-dc86-4217-b126-b807feeabecd","Type":"ContainerStarted","Data":"1df92991b6efde54a0b0dbfdeba780ae31ee4cad4c35f3b68327889ffa50f414"}
Sep 04 23:48:52.082862 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:52.082842    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:48:52.099342 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:52.099291    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn" podStartSLOduration=2.692009942 podStartE2EDuration="3.099275098s" totalImagesPullingTime="407.265156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:48:52.098338982 +0000 UTC m=+744.345006071" watchObservedRunningTime="2026-09-04 23:48:52.099275098 +0000 UTC m=+744.345942232"
Sep 04 23:48:58.093365 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:48:58.093331    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:49:09.497906 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:09.497861    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"]
Sep 04 23:49:09.498348 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:09.498089    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz" podUID="9923ebd1-57fe-4100-87dd-2c293fb22fd4" containerName="maas-api" containerID="cri-o://46dae7dae094e3d84f7ed24f447be6c2331a4a66a395593745074ddac7859065" gracePeriod=30
Sep 04 23:49:09.771769 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:09.771741    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:49:09.828941 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:09.828906    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64hrb\" (UniqueName: \"kubernetes.io/projected/9923ebd1-57fe-4100-87dd-2c293fb22fd4-kube-api-access-64hrb\") pod \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") "
Sep 04 23:49:09.829135 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:09.828977    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9923ebd1-57fe-4100-87dd-2c293fb22fd4-maas-api-tls\") pod \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") "
Sep 04 23:49:09.829135 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:09.829012    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9923ebd1-57fe-4100-87dd-2c293fb22fd4-metrics-certs\") pod \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") "
Sep 04 23:49:09.829135 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:09.829070    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9923ebd1-57fe-4100-87dd-2c293fb22fd4-openshift-service-ca\") pod \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\" (UID: \"9923ebd1-57fe-4100-87dd-2c293fb22fd4\") "
Sep 04 23:49:09.829478 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:09.829453    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9923ebd1-57fe-4100-87dd-2c293fb22fd4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9923ebd1-57fe-4100-87dd-2c293fb22fd4" (UID: "9923ebd1-57fe-4100-87dd-2c293fb22fd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:49:09.831193 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:09.831162    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9923ebd1-57fe-4100-87dd-2c293fb22fd4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9923ebd1-57fe-4100-87dd-2c293fb22fd4" (UID: "9923ebd1-57fe-4100-87dd-2c293fb22fd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:49:09.831283 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:09.831196    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9923ebd1-57fe-4100-87dd-2c293fb22fd4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9923ebd1-57fe-4100-87dd-2c293fb22fd4" (UID: "9923ebd1-57fe-4100-87dd-2c293fb22fd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:49:09.831283 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:09.831210    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9923ebd1-57fe-4100-87dd-2c293fb22fd4-kube-api-access-64hrb" (OuterVolumeSpecName: "kube-api-access-64hrb") pod "9923ebd1-57fe-4100-87dd-2c293fb22fd4" (UID: "9923ebd1-57fe-4100-87dd-2c293fb22fd4"). InnerVolumeSpecName "kube-api-access-64hrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:49:09.930137 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:09.930102    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64hrb\" (UniqueName: \"kubernetes.io/projected/9923ebd1-57fe-4100-87dd-2c293fb22fd4-kube-api-access-64hrb\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:49:09.930137 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:09.930134    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9923ebd1-57fe-4100-87dd-2c293fb22fd4-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:49:09.930352 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:09.930148    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9923ebd1-57fe-4100-87dd-2c293fb22fd4-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:49:09.930352 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:09.930161    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9923ebd1-57fe-4100-87dd-2c293fb22fd4-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:49:10.178357 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:10.178267    2562 generic.go:356] "Generic (PLEG): container finished" podID="9923ebd1-57fe-4100-87dd-2c293fb22fd4" containerID="46dae7dae094e3d84f7ed24f447be6c2331a4a66a395593745074ddac7859065" exitCode=0
Sep 04 23:49:10.178357 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:10.178317    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"
Sep 04 23:49:10.178699 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:10.178353    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz" event={"ID":"9923ebd1-57fe-4100-87dd-2c293fb22fd4","Type":"ContainerDied","Data":"46dae7dae094e3d84f7ed24f447be6c2331a4a66a395593745074ddac7859065"}
Sep 04 23:49:10.178699 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:10.178411    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz" event={"ID":"9923ebd1-57fe-4100-87dd-2c293fb22fd4","Type":"ContainerDied","Data":"4a66d2038020c3f65232bf9c103f1d1b67422674bab85249027dfbd9a4ada8b3"}
Sep 04 23:49:10.178699 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:10.178436    2562 scope.go:117] "RemoveContainer" containerID="46dae7dae094e3d84f7ed24f447be6c2331a4a66a395593745074ddac7859065"
Sep 04 23:49:10.188125 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:10.188107    2562 scope.go:117] "RemoveContainer" containerID="46dae7dae094e3d84f7ed24f447be6c2331a4a66a395593745074ddac7859065"
Sep 04 23:49:10.188412 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:49:10.188391    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46dae7dae094e3d84f7ed24f447be6c2331a4a66a395593745074ddac7859065\": container with ID starting with 46dae7dae094e3d84f7ed24f447be6c2331a4a66a395593745074ddac7859065 not found: ID does not exist" containerID="46dae7dae094e3d84f7ed24f447be6c2331a4a66a395593745074ddac7859065"
Sep 04 23:49:10.188456 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:10.188421    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46dae7dae094e3d84f7ed24f447be6c2331a4a66a395593745074ddac7859065"} err="failed to get container status \"46dae7dae094e3d84f7ed24f447be6c2331a4a66a395593745074ddac7859065\": rpc error: code = NotFound desc = could not find container \"46dae7dae094e3d84f7ed24f447be6c2331a4a66a395593745074ddac7859065\": container with ID starting with 46dae7dae094e3d84f7ed24f447be6c2331a4a66a395593745074ddac7859065 not found: ID does not exist"
Sep 04 23:49:10.199254 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:10.199222    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"]
Sep 04 23:49:10.201112 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:10.201091    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eef2d470-57675df759-n42gz"]
Sep 04 23:49:10.359101 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:10.359069    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9923ebd1-57fe-4100-87dd-2c293fb22fd4" path="/var/lib/kubelet/pods/9923ebd1-57fe-4100-87dd-2c293fb22fd4/volumes"
Sep 04 23:49:13.198715 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.198685    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"]
Sep 04 23:49:13.199333 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.199317    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9923ebd1-57fe-4100-87dd-2c293fb22fd4" containerName="maas-api"
Sep 04 23:49:13.199333 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.199336    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9923ebd1-57fe-4100-87dd-2c293fb22fd4" containerName="maas-api"
Sep 04 23:49:13.199480 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.199463    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9923ebd1-57fe-4100-87dd-2c293fb22fd4" containerName="maas-api"
Sep 04 23:49:13.204334 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.204312    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:13.207079 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.207033    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8a8c53fa-serving-cert\""
Sep 04 23:49:13.214271 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.213697    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"]
Sep 04 23:49:13.263549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.263516    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpcjg\" (UniqueName: \"kubernetes.io/projected/0b375354-426e-462c-9e62-943907b28f56-kube-api-access-rpcjg\") pod \"maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:13.263738 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.263577    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b375354-426e-462c-9e62-943907b28f56-maas-api-tls\") pod \"maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:13.263738 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.263688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b375354-426e-462c-9e62-943907b28f56-metrics-certs\") pod \"maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:13.263854 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.263738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b375354-426e-462c-9e62-943907b28f56-openshift-service-ca\") pod \"maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:13.364514 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.364474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b375354-426e-462c-9e62-943907b28f56-maas-api-tls\") pod \"maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:13.364730 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.364540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b375354-426e-462c-9e62-943907b28f56-metrics-certs\") pod \"maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:13.364730 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.364583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b375354-426e-462c-9e62-943907b28f56-openshift-service-ca\") pod \"maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:13.364730 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.364631    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpcjg\" (UniqueName: \"kubernetes.io/projected/0b375354-426e-462c-9e62-943907b28f56-kube-api-access-rpcjg\") pod \"maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:13.364730 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:49:13.364639    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-serving-cert: secret "maas-api-e2e-migrate-8a8c53fa-serving-cert" not found
Sep 04 23:49:13.364904 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:49:13.364733    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b375354-426e-462c-9e62-943907b28f56-maas-api-tls podName:0b375354-426e-462c-9e62-943907b28f56 nodeName:}" failed. No retries permitted until 2026-09-04 23:49:13.864715921 +0000 UTC m=+766.111382978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b375354-426e-462c-9e62-943907b28f56-maas-api-tls") pod "maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9" (UID: "0b375354-426e-462c-9e62-943907b28f56") : secret "maas-api-e2e-migrate-8a8c53fa-serving-cert" not found
Sep 04 23:49:13.365406 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.365381    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b375354-426e-462c-9e62-943907b28f56-openshift-service-ca\") pod \"maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:13.367108 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.367086    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b375354-426e-462c-9e62-943907b28f56-metrics-certs\") pod \"maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:13.372922 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.372899    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpcjg\" (UniqueName: \"kubernetes.io/projected/0b375354-426e-462c-9e62-943907b28f56-kube-api-access-rpcjg\") pod \"maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:13.870692 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.870633    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b375354-426e-462c-9e62-943907b28f56-maas-api-tls\") pod \"maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:13.873248 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:13.873219    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b375354-426e-462c-9e62-943907b28f56-maas-api-tls\") pod \"maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:14.119853 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:14.119781    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:14.399194 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:14.399166    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"]
Sep 04 23:49:14.400741 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:49:14.400710    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b375354_426e_462c_9e62_943907b28f56.slice/crio-42fa4856c1aa485224850bf1ef6db916ea6692cd2ef77cda80771253efc9a07c WatchSource:0}: Error finding container 42fa4856c1aa485224850bf1ef6db916ea6692cd2ef77cda80771253efc9a07c: Status 404 returned error can't find the container with id 42fa4856c1aa485224850bf1ef6db916ea6692cd2ef77cda80771253efc9a07c
Sep 04 23:49:15.208714 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:15.208681    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9" event={"ID":"0b375354-426e-462c-9e62-943907b28f56","Type":"ContainerStarted","Data":"42fa4856c1aa485224850bf1ef6db916ea6692cd2ef77cda80771253efc9a07c"}
Sep 04 23:49:16.215520 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:16.215486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9" event={"ID":"0b375354-426e-462c-9e62-943907b28f56","Type":"ContainerStarted","Data":"3df6d7c26612b606c4c3054bd156ea6532382e81329ec1b27f9eee330692c8df"}
Sep 04 23:49:16.215936 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:16.215551    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:16.233376 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:16.233333    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9" podStartSLOduration=1.920521395 podStartE2EDuration="3.233318446s" totalImagesPullingTime="1.312797051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:49:16.231692826 +0000 UTC m=+768.478359904" watchObservedRunningTime="2026-09-04 23:49:16.233318446 +0000 UTC m=+768.479985524"
Sep 04 23:49:22.228747 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:22.228718    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:44.660046 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:44.660009    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"]
Sep 04 23:49:44.660635 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:44.660371    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9" podUID="0b375354-426e-462c-9e62-943907b28f56" containerName="maas-api" containerID="cri-o://3df6d7c26612b606c4c3054bd156ea6532382e81329ec1b27f9eee330692c8df" gracePeriod=30
Sep 04 23:49:44.921400 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:44.921331    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:44.973574 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:44.973535    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b375354-426e-462c-9e62-943907b28f56-metrics-certs\") pod \"0b375354-426e-462c-9e62-943907b28f56\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") "
Sep 04 23:49:44.973574 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:44.973580    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b375354-426e-462c-9e62-943907b28f56-maas-api-tls\") pod \"0b375354-426e-462c-9e62-943907b28f56\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") "
Sep 04 23:49:44.973868 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:44.973721    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpcjg\" (UniqueName: \"kubernetes.io/projected/0b375354-426e-462c-9e62-943907b28f56-kube-api-access-rpcjg\") pod \"0b375354-426e-462c-9e62-943907b28f56\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") "
Sep 04 23:49:44.973868 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:44.973750    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b375354-426e-462c-9e62-943907b28f56-openshift-service-ca\") pod \"0b375354-426e-462c-9e62-943907b28f56\" (UID: \"0b375354-426e-462c-9e62-943907b28f56\") "
Sep 04 23:49:44.974457 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:44.974409    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b375354-426e-462c-9e62-943907b28f56-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b375354-426e-462c-9e62-943907b28f56" (UID: "0b375354-426e-462c-9e62-943907b28f56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:49:44.977136 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:44.977106    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b375354-426e-462c-9e62-943907b28f56-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0b375354-426e-462c-9e62-943907b28f56" (UID: "0b375354-426e-462c-9e62-943907b28f56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:49:44.978336 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:44.978307    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b375354-426e-462c-9e62-943907b28f56-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b375354-426e-462c-9e62-943907b28f56" (UID: "0b375354-426e-462c-9e62-943907b28f56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:49:44.978457 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:44.978329    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b375354-426e-462c-9e62-943907b28f56-kube-api-access-rpcjg" (OuterVolumeSpecName: "kube-api-access-rpcjg") pod "0b375354-426e-462c-9e62-943907b28f56" (UID: "0b375354-426e-462c-9e62-943907b28f56"). InnerVolumeSpecName "kube-api-access-rpcjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:49:45.075046 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.075007    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpcjg\" (UniqueName: \"kubernetes.io/projected/0b375354-426e-462c-9e62-943907b28f56-kube-api-access-rpcjg\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:49:45.075046 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.075039    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b375354-426e-462c-9e62-943907b28f56-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:49:45.075046 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.075048    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b375354-426e-462c-9e62-943907b28f56-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:49:45.075262 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.075057    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b375354-426e-462c-9e62-943907b28f56-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:49:45.376055 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.376018    2562 generic.go:356] "Generic (PLEG): container finished" podID="0b375354-426e-462c-9e62-943907b28f56" containerID="3df6d7c26612b606c4c3054bd156ea6532382e81329ec1b27f9eee330692c8df" exitCode=0
Sep 04 23:49:45.376216 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.376060    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"
Sep 04 23:49:45.376216 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.376111    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9" event={"ID":"0b375354-426e-462c-9e62-943907b28f56","Type":"ContainerDied","Data":"3df6d7c26612b606c4c3054bd156ea6532382e81329ec1b27f9eee330692c8df"}
Sep 04 23:49:45.376216 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.376160    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9" event={"ID":"0b375354-426e-462c-9e62-943907b28f56","Type":"ContainerDied","Data":"42fa4856c1aa485224850bf1ef6db916ea6692cd2ef77cda80771253efc9a07c"}
Sep 04 23:49:45.376216 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.376184    2562 scope.go:117] "RemoveContainer" containerID="3df6d7c26612b606c4c3054bd156ea6532382e81329ec1b27f9eee330692c8df"
Sep 04 23:49:45.385475 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.385457    2562 scope.go:117] "RemoveContainer" containerID="3df6d7c26612b606c4c3054bd156ea6532382e81329ec1b27f9eee330692c8df"
Sep 04 23:49:45.385809 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:49:45.385789    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df6d7c26612b606c4c3054bd156ea6532382e81329ec1b27f9eee330692c8df\": container with ID starting with 3df6d7c26612b606c4c3054bd156ea6532382e81329ec1b27f9eee330692c8df not found: ID does not exist" containerID="3df6d7c26612b606c4c3054bd156ea6532382e81329ec1b27f9eee330692c8df"
Sep 04 23:49:45.385874 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.385817    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df6d7c26612b606c4c3054bd156ea6532382e81329ec1b27f9eee330692c8df"} err="failed to get container status \"3df6d7c26612b606c4c3054bd156ea6532382e81329ec1b27f9eee330692c8df\": rpc error: code = NotFound desc = could not find container \"3df6d7c26612b606c4c3054bd156ea6532382e81329ec1b27f9eee330692c8df\": container with ID starting with 3df6d7c26612b606c4c3054bd156ea6532382e81329ec1b27f9eee330692c8df not found: ID does not exist"
Sep 04 23:49:45.396791 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.396763    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"]
Sep 04 23:49:45.402806 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.402774    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8a8c53fa-76558859b4-chlv9"]
Sep 04 23:49:45.679996 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.679912    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"]
Sep 04 23:49:45.680549 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.680528    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b375354-426e-462c-9e62-943907b28f56" containerName="maas-api"
Sep 04 23:49:45.680638 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.680553    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b375354-426e-462c-9e62-943907b28f56" containerName="maas-api"
Sep 04 23:49:45.680735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.680667    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b375354-426e-462c-9e62-943907b28f56" containerName="maas-api"
Sep 04 23:49:45.685206 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.685183    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.687451 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.687433    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b52c7b74-openshift-default-dockercfg-lrpsf\""
Sep 04 23:49:45.693641 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.693613    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"]
Sep 04 23:49:45.783502 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.783465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-data\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.783709 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.783520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-token\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.783709 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.783575    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-credential-socket\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.783709 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.783676    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7f937f-cae7-48de-a319-5486e2917dc2-istiod-ca-cert\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.783912 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.783735    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-podinfo\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.783912 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.783798    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-workload-certs\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.783912 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.783830    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-envoy\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.783912 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.783888    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-workload-socket\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.784094 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.783921    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2chn\" (UniqueName: \"kubernetes.io/projected/dd7f937f-cae7-48de-a319-5486e2917dc2-kube-api-access-b2chn\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.884759 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.884720    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-token\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.884759 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.884759    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-credential-socket\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.884979 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.884798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7f937f-cae7-48de-a319-5486e2917dc2-istiod-ca-cert\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.884979 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.884833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-podinfo\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.884979 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.884885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-workload-certs\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.885125 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.885070    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-envoy\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.885182 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.885144    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-workload-socket\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.885237 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.885185    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2chn\" (UniqueName: \"kubernetes.io/projected/dd7f937f-cae7-48de-a319-5486e2917dc2-kube-api-access-b2chn\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.885292 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.885253    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-data\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.885348 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.885251    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-credential-socket\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.885405 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.885342    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-workload-certs\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.885517 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.885489    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-workload-socket\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.885702 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.885609    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7f937f-cae7-48de-a319-5486e2917dc2-istiod-ca-cert\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.885771 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.885734    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-data\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.887268 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.887249    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-envoy\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.887647 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.887632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-podinfo\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.892212 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.892185    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-token\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.892299 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.892220    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2chn\" (UniqueName: \"kubernetes.io/projected/dd7f937f-cae7-48de-a319-5486e2917dc2-kube-api-access-b2chn\") pod \"e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") " pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:45.998964 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:45.998930    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:46.276679 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:46.276636    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"]
Sep 04 23:49:46.277793 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:49:46.277760    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7f937f_cae7_48de_a319_5486e2917dc2.slice/crio-47a17cbef62fcb0df8e9493165c1dd031f1d0fe0bb145d14bd1d3b38097885d5 WatchSource:0}: Error finding container 47a17cbef62fcb0df8e9493165c1dd031f1d0fe0bb145d14bd1d3b38097885d5: Status 404 returned error can't find the container with id 47a17cbef62fcb0df8e9493165c1dd031f1d0fe0bb145d14bd1d3b38097885d5
Sep 04 23:49:46.280399 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:46.280370    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:49:46.280465 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:46.280436    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:49:46.280502 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:46.280467    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:49:46.358475 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:46.358447    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b375354-426e-462c-9e62-943907b28f56" path="/var/lib/kubelet/pods/0b375354-426e-462c-9e62-943907b28f56/volumes"
Sep 04 23:49:46.387256 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:46.387219    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2" event={"ID":"dd7f937f-cae7-48de-a319-5486e2917dc2","Type":"ContainerStarted","Data":"d6ad0f0f4beb37f6820f57da261eaa3115eac3baa0f2ad8b9ab94d1c4c1ddc87"}
Sep 04 23:49:46.387442 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:46.387271    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2" event={"ID":"dd7f937f-cae7-48de-a319-5486e2917dc2","Type":"ContainerStarted","Data":"47a17cbef62fcb0df8e9493165c1dd031f1d0fe0bb145d14bd1d3b38097885d5"}
Sep 04 23:49:46.406867 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:46.406800    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2" podStartSLOduration=1.406773166 podStartE2EDuration="1.406773166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:49:46.405600012 +0000 UTC m=+798.652267095" watchObservedRunningTime="2026-09-04 23:49:46.406773166 +0000 UTC m=+798.653440246"
Sep 04 23:49:47.001153 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:47.001095    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:47.006702 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:47.006675    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:47.392551 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:47.392470    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:47.393366 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:47.393349    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:49:48.904882 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:48.904849    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"]
Sep 04 23:49:48.909603 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:48.909572    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:49:48.911935 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:48.911901    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b52c7b74-serving-cert\""
Sep 04 23:49:48.919048 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:48.919021    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"]
Sep 04 23:49:49.014797 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:49.014754    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfw4q\" (UniqueName: \"kubernetes.io/projected/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-kube-api-access-dfw4q\") pod \"maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:49:49.015123 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:49.015104    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-maas-api-tls\") pod \"maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:49:49.015238 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:49.015226    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-metrics-certs\") pod \"maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:49:49.015349 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:49.015337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-openshift-service-ca\") pod \"maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:49:49.116385 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:49.116353    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfw4q\" (UniqueName: \"kubernetes.io/projected/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-kube-api-access-dfw4q\") pod \"maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:49:49.116563 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:49.116415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-maas-api-tls\") pod \"maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:49:49.116563 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:49.116439    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-metrics-certs\") pod \"maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:49:49.116563 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:49.116474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-openshift-service-ca\") pod \"maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:49:49.117180 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:49.117159    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-openshift-service-ca\") pod \"maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:49:49.119042 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:49.119013    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-maas-api-tls\") pod \"maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:49:49.119042 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:49.119032    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-metrics-certs\") pod \"maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:49:49.124745 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:49.124718    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfw4q\" (UniqueName: \"kubernetes.io/projected/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-kube-api-access-dfw4q\") pod \"maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:49:49.227276 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:49.227244    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:49:49.523757 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:49.523730    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"]
Sep 04 23:49:49.525530 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:49:49.525507    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b10cda7_a3c4_4868_8ec6_5d6583eb9ea6.slice/crio-05130a06588efe6ce22070cb5fc701ab11bed5bcd22faa4fadaeeaf76cfc6eaf WatchSource:0}: Error finding container 05130a06588efe6ce22070cb5fc701ab11bed5bcd22faa4fadaeeaf76cfc6eaf: Status 404 returned error can't find the container with id 05130a06588efe6ce22070cb5fc701ab11bed5bcd22faa4fadaeeaf76cfc6eaf
Sep 04 23:49:50.409310 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:50.409215    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh" event={"ID":"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6","Type":"ContainerStarted","Data":"b440df059e3c85f209266d83715c54324dc80884f0a896ba07ef126887cb5ab6"}
Sep 04 23:49:50.409310 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:50.409276    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:49:50.409310 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:50.409293    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh" event={"ID":"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6","Type":"ContainerStarted","Data":"05130a06588efe6ce22070cb5fc701ab11bed5bcd22faa4fadaeeaf76cfc6eaf"}
Sep 04 23:49:50.423959 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:50.423903    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh" podStartSLOduration=1.995499608 podStartE2EDuration="2.423883435s" totalImagesPullingTime="428.383827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:49:50.423264156 +0000 UTC m=+802.669931232" watchObservedRunningTime="2026-09-04 23:49:50.423883435 +0000 UTC m=+802.670550517"
Sep 04 23:49:56.420361 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:49:56.420333    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:50:03.433827 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:03.433783    2562 patch_prober.go:28] interesting pod/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 04 23:50:03.434294 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:03.433867    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2" podUID="dd7f937f-cae7-48de-a319-5486e2917dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 04 23:50:06.512188 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:06.512152    2562 generic.go:356] "Generic (PLEG): container finished" podID="dd7f937f-cae7-48de-a319-5486e2917dc2" containerID="d6ad0f0f4beb37f6820f57da261eaa3115eac3baa0f2ad8b9ab94d1c4c1ddc87" exitCode=0
Sep 04 23:50:06.512751 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:06.512228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2" event={"ID":"dd7f937f-cae7-48de-a319-5486e2917dc2","Type":"ContainerDied","Data":"d6ad0f0f4beb37f6820f57da261eaa3115eac3baa0f2ad8b9ab94d1c4c1ddc87"}
Sep 04 23:50:06.512819 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:06.512779    2562 scope.go:117] "RemoveContainer" containerID="d6ad0f0f4beb37f6820f57da261eaa3115eac3baa0f2ad8b9ab94d1c4c1ddc87"
Sep 04 23:50:06.513888 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:06.513860    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:50:06.513999 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:06.513944    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:50:06.513999 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:06.513991    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:50:07.522876 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:07.522832    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2" event={"ID":"dd7f937f-cae7-48de-a319-5486e2917dc2","Type":"ContainerStarted","Data":"bb283e5a9c7291f6afd82d2c4abcdee2e849fac36c10dbb2689dcedfc177229b"}
Sep 04 23:50:07.999849 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:07.999810    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:50:08.000087 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:08.000064    2562 patch_prober.go:28] interesting pod/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 04 23:50:08.000140 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:08.000117    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2" podUID="dd7f937f-cae7-48de-a319-5486e2917dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 04 23:50:09.000011 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:08.999972    2562 patch_prober.go:28] interesting pod/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 04 23:50:09.000487 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:09.000045    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2" podUID="dd7f937f-cae7-48de-a319-5486e2917dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 04 23:50:09.999859 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:09.999817    2562 patch_prober.go:28] interesting pod/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 04 23:50:10.000041 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:09.999891    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2" podUID="dd7f937f-cae7-48de-a319-5486e2917dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 04 23:50:10.999512 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:10.999470    2562 patch_prober.go:28] interesting pod/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 04 23:50:10.999710 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:10.999541    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2" podUID="dd7f937f-cae7-48de-a319-5486e2917dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 04 23:50:12.000056 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:12.000018    2562 patch_prober.go:28] interesting pod/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 04 23:50:12.000454 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:12.000075    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2" podUID="dd7f937f-cae7-48de-a319-5486e2917dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 04 23:50:13.003033 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:13.003000    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:50:13.003409 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:13.003280    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:50:13.003991 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:13.003973    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:50:19.366131 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.366090    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"]
Sep 04 23:50:19.366731 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.366336    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh" podUID="6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6" containerName="maas-api" containerID="cri-o://b440df059e3c85f209266d83715c54324dc80884f0a896ba07ef126887cb5ab6" gracePeriod=30
Sep 04 23:50:19.599829 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.599796    2562 generic.go:356] "Generic (PLEG): container finished" podID="6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6" containerID="b440df059e3c85f209266d83715c54324dc80884f0a896ba07ef126887cb5ab6" exitCode=0
Sep 04 23:50:19.599980 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.599837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh" event={"ID":"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6","Type":"ContainerDied","Data":"b440df059e3c85f209266d83715c54324dc80884f0a896ba07ef126887cb5ab6"}
Sep 04 23:50:19.599980 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.599888    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh" event={"ID":"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6","Type":"ContainerDied","Data":"05130a06588efe6ce22070cb5fc701ab11bed5bcd22faa4fadaeeaf76cfc6eaf"}
Sep 04 23:50:19.599980 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.599908    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05130a06588efe6ce22070cb5fc701ab11bed5bcd22faa4fadaeeaf76cfc6eaf"
Sep 04 23:50:19.605342 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.605322    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:50:19.646618 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.646542    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-openshift-service-ca\") pod \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") "
Sep 04 23:50:19.646776 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.646630    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-maas-api-tls\") pod \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") "
Sep 04 23:50:19.646776 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.646734    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfw4q\" (UniqueName: \"kubernetes.io/projected/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-kube-api-access-dfw4q\") pod \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") "
Sep 04 23:50:19.646776 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.646770    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-metrics-certs\") pod \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\" (UID: \"6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6\") "
Sep 04 23:50:19.646982 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.646959    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6" (UID: "6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:50:19.647132 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.647111    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:19.648810 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.648783    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-kube-api-access-dfw4q" (OuterVolumeSpecName: "kube-api-access-dfw4q") pod "6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6" (UID: "6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6"). InnerVolumeSpecName "kube-api-access-dfw4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:50:19.648909 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.648819    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6" (UID: "6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:50:19.649117 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.649101    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6" (UID: "6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:50:19.747764 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.747724    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:19.747764 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.747758    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfw4q\" (UniqueName: \"kubernetes.io/projected/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-kube-api-access-dfw4q\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:19.747977 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:19.747774    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:20.605181 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:20.605140    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"]
Sep 04 23:50:20.605638 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:20.605391    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2" podUID="dd7f937f-cae7-48de-a319-5486e2917dc2" containerName="istio-proxy" containerID="cri-o://bb283e5a9c7291f6afd82d2c4abcdee2e849fac36c10dbb2689dcedfc177229b" gracePeriod=30
Sep 04 23:50:20.607172 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:20.607152    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"
Sep 04 23:50:20.634947 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:20.634867    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"]
Sep 04 23:50:20.638167 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:20.638143    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b52c7b74-6cd4d5f48-x22zh"]
Sep 04 23:50:22.359278 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:22.359233    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6" path="/var/lib/kubelet/pods/6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6/volumes"
Sep 04 23:50:24.513630 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.513599    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"]
Sep 04 23:50:24.514794 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.514762    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6" containerName="maas-api"
Sep 04 23:50:24.514794 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.514793    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6" containerName="maas-api"
Sep 04 23:50:24.514948 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.514898    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b10cda7-a3c4-4868-8ec6-5d6583eb9ea6" containerName="maas-api"
Sep 04 23:50:24.518267 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.518249    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:24.523663 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.523630    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-58fce2d8-serving-cert\""
Sep 04 23:50:24.529573 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.529547    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"]
Sep 04 23:50:24.597954 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.597923    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de683466-c93a-466f-8c86-42b55d1f667d-openshift-service-ca\") pod \"maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:24.597954 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.597959    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcfp4\" (UniqueName: \"kubernetes.io/projected/de683466-c93a-466f-8c86-42b55d1f667d-kube-api-access-fcfp4\") pod \"maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:24.598174 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.598068    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de683466-c93a-466f-8c86-42b55d1f667d-maas-api-tls\") pod \"maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:24.598174 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.598099    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de683466-c93a-466f-8c86-42b55d1f667d-metrics-certs\") pod \"maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:24.699422 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.699385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de683466-c93a-466f-8c86-42b55d1f667d-openshift-service-ca\") pod \"maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:24.699422 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.699425    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcfp4\" (UniqueName: \"kubernetes.io/projected/de683466-c93a-466f-8c86-42b55d1f667d-kube-api-access-fcfp4\") pod \"maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:24.699758 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.699528    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de683466-c93a-466f-8c86-42b55d1f667d-maas-api-tls\") pod \"maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:24.699758 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.699569    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de683466-c93a-466f-8c86-42b55d1f667d-metrics-certs\") pod \"maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:24.699758 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:50:24.699710    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-serving-cert: secret "maas-api-e2e-derive-58fce2d8-serving-cert" not found
Sep 04 23:50:24.699927 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:50:24.699804    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de683466-c93a-466f-8c86-42b55d1f667d-maas-api-tls podName:de683466-c93a-466f-8c86-42b55d1f667d nodeName:}" failed. No retries permitted until 2026-09-04 23:50:25.199782589 +0000 UTC m=+837.446449655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de683466-c93a-466f-8c86-42b55d1f667d-maas-api-tls") pod "maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz" (UID: "de683466-c93a-466f-8c86-42b55d1f667d") : secret "maas-api-e2e-derive-58fce2d8-serving-cert" not found
Sep 04 23:50:24.700203 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.700182    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de683466-c93a-466f-8c86-42b55d1f667d-openshift-service-ca\") pod \"maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:24.702108 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.702090    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de683466-c93a-466f-8c86-42b55d1f667d-metrics-certs\") pod \"maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:24.708820 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:24.708796    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcfp4\" (UniqueName: \"kubernetes.io/projected/de683466-c93a-466f-8c86-42b55d1f667d-kube-api-access-fcfp4\") pod \"maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:25.204234 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.204194    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de683466-c93a-466f-8c86-42b55d1f667d-maas-api-tls\") pod \"maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:25.206687 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.206643    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de683466-c93a-466f-8c86-42b55d1f667d-maas-api-tls\") pod \"maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:25.435006 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.434967    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:25.707636 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.707607    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"]
Sep 04 23:50:25.709056 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:50:25.709021    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde683466_c93a_466f_8c86_42b55d1f667d.slice/crio-9076d3c6fb7d7f7f0173bd526a0ac0ce532931c9741600386cd3b65f1f2411ab WatchSource:0}: Error finding container 9076d3c6fb7d7f7f0173bd526a0ac0ce532931c9741600386cd3b65f1f2411ab: Status 404 returned error can't find the container with id 9076d3c6fb7d7f7f0173bd526a0ac0ce532931c9741600386cd3b65f1f2411ab
Sep 04 23:50:25.863077 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.863052    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:50:25.911919 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.911866    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2chn\" (UniqueName: \"kubernetes.io/projected/dd7f937f-cae7-48de-a319-5486e2917dc2-kube-api-access-b2chn\") pod \"dd7f937f-cae7-48de-a319-5486e2917dc2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") "
Sep 04 23:50:25.911919 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.911923    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-workload-certs\") pod \"dd7f937f-cae7-48de-a319-5486e2917dc2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") "
Sep 04 23:50:25.912152 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.911963    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-workload-socket\") pod \"dd7f937f-cae7-48de-a319-5486e2917dc2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") "
Sep 04 23:50:25.912152 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.911990    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-credential-socket\") pod \"dd7f937f-cae7-48de-a319-5486e2917dc2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") "
Sep 04 23:50:25.912152 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.912031    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-token\") pod \"dd7f937f-cae7-48de-a319-5486e2917dc2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") "
Sep 04 23:50:25.912152 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.912064    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-envoy\") pod \"dd7f937f-cae7-48de-a319-5486e2917dc2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") "
Sep 04 23:50:25.912152 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.912120    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7f937f-cae7-48de-a319-5486e2917dc2-istiod-ca-cert\") pod \"dd7f937f-cae7-48de-a319-5486e2917dc2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") "
Sep 04 23:50:25.912403 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.912160    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-data\") pod \"dd7f937f-cae7-48de-a319-5486e2917dc2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") "
Sep 04 23:50:25.912403 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.912187    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-podinfo\") pod \"dd7f937f-cae7-48de-a319-5486e2917dc2\" (UID: \"dd7f937f-cae7-48de-a319-5486e2917dc2\") "
Sep 04 23:50:25.912403 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.912243    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dd7f937f-cae7-48de-a319-5486e2917dc2" (UID: "dd7f937f-cae7-48de-a319-5486e2917dc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:50:25.912403 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.912320    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dd7f937f-cae7-48de-a319-5486e2917dc2" (UID: "dd7f937f-cae7-48de-a319-5486e2917dc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:50:25.912588 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.912455    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-data" (OuterVolumeSpecName: "istio-data") pod "dd7f937f-cae7-48de-a319-5486e2917dc2" (UID: "dd7f937f-cae7-48de-a319-5486e2917dc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:50:25.912588 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.912552    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7f937f-cae7-48de-a319-5486e2917dc2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dd7f937f-cae7-48de-a319-5486e2917dc2" (UID: "dd7f937f-cae7-48de-a319-5486e2917dc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:50:25.912588 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.912567    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-workload-socket\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:25.912588 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.912586    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-data\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:25.912754 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.912601    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-workload-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:25.912754 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.912388    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dd7f937f-cae7-48de-a319-5486e2917dc2" (UID: "dd7f937f-cae7-48de-a319-5486e2917dc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:50:25.914377 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.914354    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dd7f937f-cae7-48de-a319-5486e2917dc2" (UID: "dd7f937f-cae7-48de-a319-5486e2917dc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:50:25.914561 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.914535    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dd7f937f-cae7-48de-a319-5486e2917dc2" (UID: "dd7f937f-cae7-48de-a319-5486e2917dc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:50:25.914561 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.914548    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-token" (OuterVolumeSpecName: "istio-token") pod "dd7f937f-cae7-48de-a319-5486e2917dc2" (UID: "dd7f937f-cae7-48de-a319-5486e2917dc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:50:25.914735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:25.914564    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7f937f-cae7-48de-a319-5486e2917dc2-kube-api-access-b2chn" (OuterVolumeSpecName: "kube-api-access-b2chn") pod "dd7f937f-cae7-48de-a319-5486e2917dc2" (UID: "dd7f937f-cae7-48de-a319-5486e2917dc2"). InnerVolumeSpecName "kube-api-access-b2chn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:50:26.013113 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.013077    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-credential-socket\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:26.013113 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.013109    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-token\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:26.013113 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.013119    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-envoy\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:26.013113 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.013126    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7f937f-cae7-48de-a319-5486e2917dc2-istiod-ca-cert\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:26.013383 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.013134    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7f937f-cae7-48de-a319-5486e2917dc2-istio-podinfo\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:26.013383 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.013142    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2chn\" (UniqueName: \"kubernetes.io/projected/dd7f937f-cae7-48de-a319-5486e2917dc2-kube-api-access-b2chn\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:26.643429 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.643392    2562 generic.go:356] "Generic (PLEG): container finished" podID="dd7f937f-cae7-48de-a319-5486e2917dc2" containerID="bb283e5a9c7291f6afd82d2c4abcdee2e849fac36c10dbb2689dcedfc177229b" exitCode=0
Sep 04 23:50:26.643611 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.643452    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"
Sep 04 23:50:26.643611 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.643470    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2" event={"ID":"dd7f937f-cae7-48de-a319-5486e2917dc2","Type":"ContainerDied","Data":"bb283e5a9c7291f6afd82d2c4abcdee2e849fac36c10dbb2689dcedfc177229b"}
Sep 04 23:50:26.643611 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.643527    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2" event={"ID":"dd7f937f-cae7-48de-a319-5486e2917dc2","Type":"ContainerDied","Data":"47a17cbef62fcb0df8e9493165c1dd031f1d0fe0bb145d14bd1d3b38097885d5"}
Sep 04 23:50:26.643611 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.643551    2562 scope.go:117] "RemoveContainer" containerID="bb283e5a9c7291f6afd82d2c4abcdee2e849fac36c10dbb2689dcedfc177229b"
Sep 04 23:50:26.645377 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.645345    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz" event={"ID":"de683466-c93a-466f-8c86-42b55d1f667d","Type":"ContainerStarted","Data":"4e1c1a38906f9824ce29f4cb1dee5a7525b1226345a0bbe21f34efd80218136e"}
Sep 04 23:50:26.645515 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.645404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz" event={"ID":"de683466-c93a-466f-8c86-42b55d1f667d","Type":"ContainerStarted","Data":"9076d3c6fb7d7f7f0173bd526a0ac0ce532931c9741600386cd3b65f1f2411ab"}
Sep 04 23:50:26.645515 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.645481    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:26.655201 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.655180    2562 scope.go:117] "RemoveContainer" containerID="d6ad0f0f4beb37f6820f57da261eaa3115eac3baa0f2ad8b9ab94d1c4c1ddc87"
Sep 04 23:50:26.660957 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.660935    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"]
Sep 04 23:50:26.663879 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.663850    2562 scope.go:117] "RemoveContainer" containerID="bb283e5a9c7291f6afd82d2c4abcdee2e849fac36c10dbb2689dcedfc177229b"
Sep 04 23:50:26.664197 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:50:26.664170    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb283e5a9c7291f6afd82d2c4abcdee2e849fac36c10dbb2689dcedfc177229b\": container with ID starting with bb283e5a9c7291f6afd82d2c4abcdee2e849fac36c10dbb2689dcedfc177229b not found: ID does not exist" containerID="bb283e5a9c7291f6afd82d2c4abcdee2e849fac36c10dbb2689dcedfc177229b"
Sep 04 23:50:26.664314 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.664207    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb283e5a9c7291f6afd82d2c4abcdee2e849fac36c10dbb2689dcedfc177229b"} err="failed to get container status \"bb283e5a9c7291f6afd82d2c4abcdee2e849fac36c10dbb2689dcedfc177229b\": rpc error: code = NotFound desc = could not find container \"bb283e5a9c7291f6afd82d2c4abcdee2e849fac36c10dbb2689dcedfc177229b\": container with ID starting with bb283e5a9c7291f6afd82d2c4abcdee2e849fac36c10dbb2689dcedfc177229b not found: ID does not exist"
Sep 04 23:50:26.664314 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.664231    2562 scope.go:117] "RemoveContainer" containerID="d6ad0f0f4beb37f6820f57da261eaa3115eac3baa0f2ad8b9ab94d1c4c1ddc87"
Sep 04 23:50:26.664561 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:50:26.664543    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6ad0f0f4beb37f6820f57da261eaa3115eac3baa0f2ad8b9ab94d1c4c1ddc87\": container with ID starting with d6ad0f0f4beb37f6820f57da261eaa3115eac3baa0f2ad8b9ab94d1c4c1ddc87 not found: ID does not exist" containerID="d6ad0f0f4beb37f6820f57da261eaa3115eac3baa0f2ad8b9ab94d1c4c1ddc87"
Sep 04 23:50:26.664615 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.664567    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6ad0f0f4beb37f6820f57da261eaa3115eac3baa0f2ad8b9ab94d1c4c1ddc87"} err="failed to get container status \"d6ad0f0f4beb37f6820f57da261eaa3115eac3baa0f2ad8b9ab94d1c4c1ddc87\": rpc error: code = NotFound desc = could not find container \"d6ad0f0f4beb37f6820f57da261eaa3115eac3baa0f2ad8b9ab94d1c4c1ddc87\": container with ID starting with d6ad0f0f4beb37f6820f57da261eaa3115eac3baa0f2ad8b9ab94d1c4c1ddc87 not found: ID does not exist"
Sep 04 23:50:26.666470 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.666453    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b52c7b74-openshift-default-6ff75bdb48-v44r2"]
Sep 04 23:50:26.679341 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:26.679302    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz" podStartSLOduration=2.245888718 podStartE2EDuration="2.679288953s" totalImagesPullingTime="433.400235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:50:26.678430363 +0000 UTC m=+838.925097441" watchObservedRunningTime="2026-09-04 23:50:26.679288953 +0000 UTC m=+838.925956032"
Sep 04 23:50:28.358916 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:28.358885    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd7f937f-cae7-48de-a319-5486e2917dc2" path="/var/lib/kubelet/pods/dd7f937f-cae7-48de-a319-5486e2917dc2/volumes"
Sep 04 23:50:32.661571 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:32.661541    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:54.875076 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:54.875041    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"]
Sep 04 23:50:54.875638 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:54.875275    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz" podUID="de683466-c93a-466f-8c86-42b55d1f667d" containerName="maas-api" containerID="cri-o://4e1c1a38906f9824ce29f4cb1dee5a7525b1226345a0bbe21f34efd80218136e" gracePeriod=30
Sep 04 23:50:55.013569 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.013530    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"]
Sep 04 23:50:55.013804 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.013767    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl" podUID="0032a5be-72f4-465a-a5a9-3b09880ae938" containerName="maas-api" containerID="cri-o://374f83cdd3783146817a5282752b96659a251367b7a2deea0ecddb72be0eadb3" gracePeriod=30
Sep 04 23:50:55.157082 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.157054    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:55.209900 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.209768    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de683466-c93a-466f-8c86-42b55d1f667d-openshift-service-ca\") pod \"de683466-c93a-466f-8c86-42b55d1f667d\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") "
Sep 04 23:50:55.209900 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.209868    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de683466-c93a-466f-8c86-42b55d1f667d-metrics-certs\") pod \"de683466-c93a-466f-8c86-42b55d1f667d\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") "
Sep 04 23:50:55.209900 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.209901    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcfp4\" (UniqueName: \"kubernetes.io/projected/de683466-c93a-466f-8c86-42b55d1f667d-kube-api-access-fcfp4\") pod \"de683466-c93a-466f-8c86-42b55d1f667d\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") "
Sep 04 23:50:55.210595 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.209936    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de683466-c93a-466f-8c86-42b55d1f667d-maas-api-tls\") pod \"de683466-c93a-466f-8c86-42b55d1f667d\" (UID: \"de683466-c93a-466f-8c86-42b55d1f667d\") "
Sep 04 23:50:55.211117 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.211086    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de683466-c93a-466f-8c86-42b55d1f667d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "de683466-c93a-466f-8c86-42b55d1f667d" (UID: "de683466-c93a-466f-8c86-42b55d1f667d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:50:55.214527 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.214487    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de683466-c93a-466f-8c86-42b55d1f667d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "de683466-c93a-466f-8c86-42b55d1f667d" (UID: "de683466-c93a-466f-8c86-42b55d1f667d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:50:55.216223 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.216108    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de683466-c93a-466f-8c86-42b55d1f667d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "de683466-c93a-466f-8c86-42b55d1f667d" (UID: "de683466-c93a-466f-8c86-42b55d1f667d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:50:55.216223 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.216187    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de683466-c93a-466f-8c86-42b55d1f667d-kube-api-access-fcfp4" (OuterVolumeSpecName: "kube-api-access-fcfp4") pod "de683466-c93a-466f-8c86-42b55d1f667d" (UID: "de683466-c93a-466f-8c86-42b55d1f667d"). InnerVolumeSpecName "kube-api-access-fcfp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:50:55.290348 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.290320    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:50:55.311318 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.311289    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0032a5be-72f4-465a-a5a9-3b09880ae938-openshift-service-ca\") pod \"0032a5be-72f4-465a-a5a9-3b09880ae938\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") "
Sep 04 23:50:55.311509 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.311332    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npcg4\" (UniqueName: \"kubernetes.io/projected/0032a5be-72f4-465a-a5a9-3b09880ae938-kube-api-access-npcg4\") pod \"0032a5be-72f4-465a-a5a9-3b09880ae938\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") "
Sep 04 23:50:55.311509 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.311382    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0032a5be-72f4-465a-a5a9-3b09880ae938-maas-api-tls\") pod \"0032a5be-72f4-465a-a5a9-3b09880ae938\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") "
Sep 04 23:50:55.311509 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.311407    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0032a5be-72f4-465a-a5a9-3b09880ae938-metrics-certs\") pod \"0032a5be-72f4-465a-a5a9-3b09880ae938\" (UID: \"0032a5be-72f4-465a-a5a9-3b09880ae938\") "
Sep 04 23:50:55.311734 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.311672    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de683466-c93a-466f-8c86-42b55d1f667d-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:55.311734 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.311694    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de683466-c93a-466f-8c86-42b55d1f667d-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:55.311734 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.311708    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcfp4\" (UniqueName: \"kubernetes.io/projected/de683466-c93a-466f-8c86-42b55d1f667d-kube-api-access-fcfp4\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:55.311734 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.311707    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0032a5be-72f4-465a-a5a9-3b09880ae938-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0032a5be-72f4-465a-a5a9-3b09880ae938" (UID: "0032a5be-72f4-465a-a5a9-3b09880ae938"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:50:55.311734 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.311721    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de683466-c93a-466f-8c86-42b55d1f667d-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:55.313576 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.313546    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0032a5be-72f4-465a-a5a9-3b09880ae938-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0032a5be-72f4-465a-a5a9-3b09880ae938" (UID: "0032a5be-72f4-465a-a5a9-3b09880ae938"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:50:55.313722 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.313588    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0032a5be-72f4-465a-a5a9-3b09880ae938-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0032a5be-72f4-465a-a5a9-3b09880ae938" (UID: "0032a5be-72f4-465a-a5a9-3b09880ae938"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:50:55.313955 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.313922    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0032a5be-72f4-465a-a5a9-3b09880ae938-kube-api-access-npcg4" (OuterVolumeSpecName: "kube-api-access-npcg4") pod "0032a5be-72f4-465a-a5a9-3b09880ae938" (UID: "0032a5be-72f4-465a-a5a9-3b09880ae938"). InnerVolumeSpecName "kube-api-access-npcg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:50:55.412394 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.412299    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0032a5be-72f4-465a-a5a9-3b09880ae938-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:55.412394 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.412333    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npcg4\" (UniqueName: \"kubernetes.io/projected/0032a5be-72f4-465a-a5a9-3b09880ae938-kube-api-access-npcg4\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:55.412394 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.412347    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0032a5be-72f4-465a-a5a9-3b09880ae938-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:55.412394 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.412359    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0032a5be-72f4-465a-a5a9-3b09880ae938-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:50:55.809527 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.809491    2562 generic.go:356] "Generic (PLEG): container finished" podID="0032a5be-72f4-465a-a5a9-3b09880ae938" containerID="374f83cdd3783146817a5282752b96659a251367b7a2deea0ecddb72be0eadb3" exitCode=0
Sep 04 23:50:55.809745 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.809538    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"
Sep 04 23:50:55.809745 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.809573    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl" event={"ID":"0032a5be-72f4-465a-a5a9-3b09880ae938","Type":"ContainerDied","Data":"374f83cdd3783146817a5282752b96659a251367b7a2deea0ecddb72be0eadb3"}
Sep 04 23:50:55.809745 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.809626    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl" event={"ID":"0032a5be-72f4-465a-a5a9-3b09880ae938","Type":"ContainerDied","Data":"575846af831f7dc919486985ea972c22313937ab7705a6fffc8a273d023a0fe5"}
Sep 04 23:50:55.809745 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.809664    2562 scope.go:117] "RemoveContainer" containerID="374f83cdd3783146817a5282752b96659a251367b7a2deea0ecddb72be0eadb3"
Sep 04 23:50:55.810987 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.810962    2562 generic.go:356] "Generic (PLEG): container finished" podID="de683466-c93a-466f-8c86-42b55d1f667d" containerID="4e1c1a38906f9824ce29f4cb1dee5a7525b1226345a0bbe21f34efd80218136e" exitCode=0
Sep 04 23:50:55.811101 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.811000    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"
Sep 04 23:50:55.811160 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.811012    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz" event={"ID":"de683466-c93a-466f-8c86-42b55d1f667d","Type":"ContainerDied","Data":"4e1c1a38906f9824ce29f4cb1dee5a7525b1226345a0bbe21f34efd80218136e"}
Sep 04 23:50:55.811205 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.811187    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz" event={"ID":"de683466-c93a-466f-8c86-42b55d1f667d","Type":"ContainerDied","Data":"9076d3c6fb7d7f7f0173bd526a0ac0ce532931c9741600386cd3b65f1f2411ab"}
Sep 04 23:50:55.828270 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.828248    2562 scope.go:117] "RemoveContainer" containerID="374f83cdd3783146817a5282752b96659a251367b7a2deea0ecddb72be0eadb3"
Sep 04 23:50:55.828839 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:50:55.828816    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374f83cdd3783146817a5282752b96659a251367b7a2deea0ecddb72be0eadb3\": container with ID starting with 374f83cdd3783146817a5282752b96659a251367b7a2deea0ecddb72be0eadb3 not found: ID does not exist" containerID="374f83cdd3783146817a5282752b96659a251367b7a2deea0ecddb72be0eadb3"
Sep 04 23:50:55.828943 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.828850    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374f83cdd3783146817a5282752b96659a251367b7a2deea0ecddb72be0eadb3"} err="failed to get container status \"374f83cdd3783146817a5282752b96659a251367b7a2deea0ecddb72be0eadb3\": rpc error: code = NotFound desc = could not find container \"374f83cdd3783146817a5282752b96659a251367b7a2deea0ecddb72be0eadb3\": container with ID starting with 374f83cdd3783146817a5282752b96659a251367b7a2deea0ecddb72be0eadb3 not found: ID does not exist"
Sep 04 23:50:55.828943 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.828875    2562 scope.go:117] "RemoveContainer" containerID="4e1c1a38906f9824ce29f4cb1dee5a7525b1226345a0bbe21f34efd80218136e"
Sep 04 23:50:55.838198 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.838180    2562 scope.go:117] "RemoveContainer" containerID="4e1c1a38906f9824ce29f4cb1dee5a7525b1226345a0bbe21f34efd80218136e"
Sep 04 23:50:55.838559 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:50:55.838523    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1c1a38906f9824ce29f4cb1dee5a7525b1226345a0bbe21f34efd80218136e\": container with ID starting with 4e1c1a38906f9824ce29f4cb1dee5a7525b1226345a0bbe21f34efd80218136e not found: ID does not exist" containerID="4e1c1a38906f9824ce29f4cb1dee5a7525b1226345a0bbe21f34efd80218136e"
Sep 04 23:50:55.838669 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.838568    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1c1a38906f9824ce29f4cb1dee5a7525b1226345a0bbe21f34efd80218136e"} err="failed to get container status \"4e1c1a38906f9824ce29f4cb1dee5a7525b1226345a0bbe21f34efd80218136e\": rpc error: code = NotFound desc = could not find container \"4e1c1a38906f9824ce29f4cb1dee5a7525b1226345a0bbe21f34efd80218136e\": container with ID starting with 4e1c1a38906f9824ce29f4cb1dee5a7525b1226345a0bbe21f34efd80218136e not found: ID does not exist"
Sep 04 23:50:55.840228 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.840208    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"]
Sep 04 23:50:55.847075 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.847049    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3c6688-b6cbdf9dc-6vlbl"]
Sep 04 23:50:55.863176 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.863153    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"]
Sep 04 23:50:55.869531 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:55.869512    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-58fce2d8-bb5fd4bbd-vtvrz"]
Sep 04 23:50:56.358716 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:56.358684    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0032a5be-72f4-465a-a5a9-3b09880ae938" path="/var/lib/kubelet/pods/0032a5be-72f4-465a-a5a9-3b09880ae938/volumes"
Sep 04 23:50:56.359160 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:50:56.359143    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de683466-c93a-466f-8c86-42b55d1f667d" path="/var/lib/kubelet/pods/de683466-c93a-466f-8c86-42b55d1f667d/volumes"
Sep 04 23:51:01.721101 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:01.721064    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"]
Sep 04 23:51:01.721862 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:01.721288    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn" podUID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerName="istio-proxy" containerID="cri-o://d6686147663bcbd9494f5baad001e4c881497e22161e7b91f7d625f640cd216d" gracePeriod=30
Sep 04 23:51:06.485624 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.485588    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"]
Sep 04 23:51:06.486302 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.486277    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de683466-c93a-466f-8c86-42b55d1f667d" containerName="maas-api"
Sep 04 23:51:06.486302 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.486304    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="de683466-c93a-466f-8c86-42b55d1f667d" containerName="maas-api"
Sep 04 23:51:06.486450 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.486339    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd7f937f-cae7-48de-a319-5486e2917dc2" containerName="istio-proxy"
Sep 04 23:51:06.486450 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.486348    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd7f937f-cae7-48de-a319-5486e2917dc2" containerName="istio-proxy"
Sep 04 23:51:06.486450 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.486361    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd7f937f-cae7-48de-a319-5486e2917dc2" containerName="istio-proxy"
Sep 04 23:51:06.486450 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.486368    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd7f937f-cae7-48de-a319-5486e2917dc2" containerName="istio-proxy"
Sep 04 23:51:06.486450 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.486393    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0032a5be-72f4-465a-a5a9-3b09880ae938" containerName="maas-api"
Sep 04 23:51:06.486450 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.486401    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0032a5be-72f4-465a-a5a9-3b09880ae938" containerName="maas-api"
Sep 04 23:51:06.486774 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.486503    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="de683466-c93a-466f-8c86-42b55d1f667d" containerName="maas-api"
Sep 04 23:51:06.486774 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.486518    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd7f937f-cae7-48de-a319-5486e2917dc2" containerName="istio-proxy"
Sep 04 23:51:06.486774 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.486527    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd7f937f-cae7-48de-a319-5486e2917dc2" containerName="istio-proxy"
Sep 04 23:51:06.486774 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.486536    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0032a5be-72f4-465a-a5a9-3b09880ae938" containerName="maas-api"
Sep 04 23:51:06.491498 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.491471    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:06.493737 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.493710    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-6fa8d6-serving-cert\""
Sep 04 23:51:06.499769 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.499731    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"]
Sep 04 23:51:06.632238 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.632178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shggw\" (UniqueName: \"kubernetes.io/projected/3cc43103-ea9a-42c2-99c4-f8688d5136b1-kube-api-access-shggw\") pod \"maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:06.632421 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.632292    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc43103-ea9a-42c2-99c4-f8688d5136b1-openshift-service-ca\") pod \"maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:06.632421 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.632341    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc43103-ea9a-42c2-99c4-f8688d5136b1-metrics-certs\") pod \"maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:06.632421 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.632381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc43103-ea9a-42c2-99c4-f8688d5136b1-maas-api-tls\") pod \"maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:06.733146 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.733099    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shggw\" (UniqueName: \"kubernetes.io/projected/3cc43103-ea9a-42c2-99c4-f8688d5136b1-kube-api-access-shggw\") pod \"maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:06.733334 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.733186    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc43103-ea9a-42c2-99c4-f8688d5136b1-openshift-service-ca\") pod \"maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:06.733334 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.733225    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc43103-ea9a-42c2-99c4-f8688d5136b1-metrics-certs\") pod \"maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:06.733334 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.733275    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc43103-ea9a-42c2-99c4-f8688d5136b1-maas-api-tls\") pod \"maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:06.733509 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:51:06.733446    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-serving-cert: secret "maas-api-e2e-explicit-6fa8d6-serving-cert" not found
Sep 04 23:51:06.733580 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:51:06.733554    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc43103-ea9a-42c2-99c4-f8688d5136b1-maas-api-tls podName:3cc43103-ea9a-42c2-99c4-f8688d5136b1 nodeName:}" failed. No retries permitted until 2026-09-04 23:51:07.233531958 +0000 UTC m=+879.480199020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3cc43103-ea9a-42c2-99c4-f8688d5136b1-maas-api-tls") pod "maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz" (UID: "3cc43103-ea9a-42c2-99c4-f8688d5136b1") : secret "maas-api-e2e-explicit-6fa8d6-serving-cert" not found
Sep 04 23:51:06.733970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.733946    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc43103-ea9a-42c2-99c4-f8688d5136b1-openshift-service-ca\") pod \"maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:06.735838 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.735780    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc43103-ea9a-42c2-99c4-f8688d5136b1-metrics-certs\") pod \"maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:06.743826 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.743799    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shggw\" (UniqueName: \"kubernetes.io/projected/3cc43103-ea9a-42c2-99c4-f8688d5136b1-kube-api-access-shggw\") pod \"maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:06.876494 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.876458    2562 generic.go:356] "Generic (PLEG): container finished" podID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerID="d6686147663bcbd9494f5baad001e4c881497e22161e7b91f7d625f640cd216d" exitCode=0
Sep 04 23:51:06.876694 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.876532    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn" event={"ID":"976dc20d-a7b7-4894-bfd6-d2c84141c97e","Type":"ContainerDied","Data":"d6686147663bcbd9494f5baad001e4c881497e22161e7b91f7d625f640cd216d"}
Sep 04 23:51:06.876694 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.876597    2562 scope.go:117] "RemoveContainer" containerID="8036f75e2036dcd7a8f307a84344d52028e313b360c351ea2c5f95f3aed1846b"
Sep 04 23:51:06.993775 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:06.993683    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:51:07.137018 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.136983    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istiod-ca-cert\") pod \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") "
Sep 04 23:51:07.137018 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137024    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-envoy\") pod \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") "
Sep 04 23:51:07.137265 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137086    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-data\") pod \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") "
Sep 04 23:51:07.137265 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137145    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lfnx\" (UniqueName: \"kubernetes.io/projected/976dc20d-a7b7-4894-bfd6-d2c84141c97e-kube-api-access-8lfnx\") pod \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") "
Sep 04 23:51:07.137265 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137179    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-podinfo\") pod \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") "
Sep 04 23:51:07.137265 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137237    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-workload-socket\") pod \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") "
Sep 04 23:51:07.137466 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137375    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-token\") pod \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") "
Sep 04 23:51:07.137466 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137430    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "976dc20d-a7b7-4894-bfd6-d2c84141c97e" (UID: "976dc20d-a7b7-4894-bfd6-d2c84141c97e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:51:07.137466 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137461    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-credential-socket\") pod \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") "
Sep 04 23:51:07.137631 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137479    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-data" (OuterVolumeSpecName: "istio-data") pod "976dc20d-a7b7-4894-bfd6-d2c84141c97e" (UID: "976dc20d-a7b7-4894-bfd6-d2c84141c97e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:51:07.137631 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137491    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "976dc20d-a7b7-4894-bfd6-d2c84141c97e" (UID: "976dc20d-a7b7-4894-bfd6-d2c84141c97e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:51:07.137631 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137495    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-workload-certs\") pod \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\" (UID: \"976dc20d-a7b7-4894-bfd6-d2c84141c97e\") "
Sep 04 23:51:07.137822 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137693    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "976dc20d-a7b7-4894-bfd6-d2c84141c97e" (UID: "976dc20d-a7b7-4894-bfd6-d2c84141c97e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:51:07.137822 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137724    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "976dc20d-a7b7-4894-bfd6-d2c84141c97e" (UID: "976dc20d-a7b7-4894-bfd6-d2c84141c97e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:51:07.137984 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137947    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-credential-socket\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:51:07.137984 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137972    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-workload-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:51:07.138107 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.137987    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istiod-ca-cert\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:51:07.138107 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.138001    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-data\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:51:07.138107 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.138013    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-workload-socket\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:51:07.139542 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.139452    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "976dc20d-a7b7-4894-bfd6-d2c84141c97e" (UID: "976dc20d-a7b7-4894-bfd6-d2c84141c97e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:51:07.139542 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.139510    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976dc20d-a7b7-4894-bfd6-d2c84141c97e-kube-api-access-8lfnx" (OuterVolumeSpecName: "kube-api-access-8lfnx") pod "976dc20d-a7b7-4894-bfd6-d2c84141c97e" (UID: "976dc20d-a7b7-4894-bfd6-d2c84141c97e"). InnerVolumeSpecName "kube-api-access-8lfnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:51:07.139742 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.139548    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-token" (OuterVolumeSpecName: "istio-token") pod "976dc20d-a7b7-4894-bfd6-d2c84141c97e" (UID: "976dc20d-a7b7-4894-bfd6-d2c84141c97e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:51:07.139742 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.139561    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "976dc20d-a7b7-4894-bfd6-d2c84141c97e" (UID: "976dc20d-a7b7-4894-bfd6-d2c84141c97e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:51:07.238903 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.238862    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc43103-ea9a-42c2-99c4-f8688d5136b1-maas-api-tls\") pod \"maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:07.239071 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.238945    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-token\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:51:07.239071 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.238955    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-envoy\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:51:07.239071 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.238964    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lfnx\" (UniqueName: \"kubernetes.io/projected/976dc20d-a7b7-4894-bfd6-d2c84141c97e-kube-api-access-8lfnx\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:51:07.239071 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.238978    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/976dc20d-a7b7-4894-bfd6-d2c84141c97e-istio-podinfo\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:51:07.241307 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.241281    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc43103-ea9a-42c2-99c4-f8688d5136b1-maas-api-tls\") pod \"maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:07.407893 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.407805    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:07.679501 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.679470    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"]
Sep 04 23:51:07.681604 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:51:07.681580    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc43103_ea9a_42c2_99c4_f8688d5136b1.slice/crio-797267716f841140a8a676681bba5adc4f603ce4d8e7bc5ab913d4298241d0b2 WatchSource:0}: Error finding container 797267716f841140a8a676681bba5adc4f603ce4d8e7bc5ab913d4298241d0b2: Status 404 returned error can't find the container with id 797267716f841140a8a676681bba5adc4f603ce4d8e7bc5ab913d4298241d0b2
Sep 04 23:51:07.882848 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.882806    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"
Sep 04 23:51:07.882848 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.882837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn" event={"ID":"976dc20d-a7b7-4894-bfd6-d2c84141c97e","Type":"ContainerDied","Data":"810a9836545717503bc2e23d4cf578b519172d49fa1221cd731ffea809dec81b"}
Sep 04 23:51:07.883085 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.882904    2562 scope.go:117] "RemoveContainer" containerID="d6686147663bcbd9494f5baad001e4c881497e22161e7b91f7d625f640cd216d"
Sep 04 23:51:07.884320 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.884266    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz" event={"ID":"3cc43103-ea9a-42c2-99c4-f8688d5136b1","Type":"ContainerStarted","Data":"797267716f841140a8a676681bba5adc4f603ce4d8e7bc5ab913d4298241d0b2"}
Sep 04 23:51:07.904857 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.904833    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"]
Sep 04 23:51:07.908743 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.908719    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn"]
Sep 04 23:51:07.984006 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.983967    2562 patch_prober.go:28] interesting pod/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 04 23:51:07.984189 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:07.984033    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-3c6688-openshift-default-595dfc6494-fgsdn" podUID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 04 23:51:08.358865 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:08.358829    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" path="/var/lib/kubelet/pods/976dc20d-a7b7-4894-bfd6-d2c84141c97e/volumes"
Sep 04 23:51:08.891070 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:08.891033    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz" event={"ID":"3cc43103-ea9a-42c2-99c4-f8688d5136b1","Type":"ContainerStarted","Data":"a7a2ce9793b48cd07cdd8e2a24ceba9cb70144723919d8163400d865160879ab"}
Sep 04 23:51:08.891470 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:08.891097    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:08.908115 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:08.908050    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz" podStartSLOduration=2.324560881 podStartE2EDuration="2.90803468s" totalImagesPullingTime="583.473799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:51:08.905808157 +0000 UTC m=+881.152475236" watchObservedRunningTime="2026-09-04 23:51:08.90803468 +0000 UTC m=+881.154701758"
Sep 04 23:51:14.903053 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:14.903020    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:51:28.238342 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:28.238260    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 04 23:51:28.239447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:28.239424    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 04 23:51:28.243027 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:28.243003    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 04 23:51:28.244161 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:51:28.244141    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 04 23:52:13.400324 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:13.400291    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"]
Sep 04 23:52:13.400882 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:13.400511    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz" podUID="3cc43103-ea9a-42c2-99c4-f8688d5136b1" containerName="maas-api" containerID="cri-o://a7a2ce9793b48cd07cdd8e2a24ceba9cb70144723919d8163400d865160879ab" gracePeriod=30
Sep 04 23:52:13.656234 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:13.656175    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:52:13.764922 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:13.764887    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc43103-ea9a-42c2-99c4-f8688d5136b1-openshift-service-ca\") pod \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") "
Sep 04 23:52:13.765120 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:13.764959    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc43103-ea9a-42c2-99c4-f8688d5136b1-maas-api-tls\") pod \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") "
Sep 04 23:52:13.765120 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:13.764983    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc43103-ea9a-42c2-99c4-f8688d5136b1-metrics-certs\") pod \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") "
Sep 04 23:52:13.765120 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:13.765059    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shggw\" (UniqueName: \"kubernetes.io/projected/3cc43103-ea9a-42c2-99c4-f8688d5136b1-kube-api-access-shggw\") pod \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\" (UID: \"3cc43103-ea9a-42c2-99c4-f8688d5136b1\") "
Sep 04 23:52:13.765524 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:13.765487    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cc43103-ea9a-42c2-99c4-f8688d5136b1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3cc43103-ea9a-42c2-99c4-f8688d5136b1" (UID: "3cc43103-ea9a-42c2-99c4-f8688d5136b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:52:13.767176 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:13.767145    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cc43103-ea9a-42c2-99c4-f8688d5136b1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3cc43103-ea9a-42c2-99c4-f8688d5136b1" (UID: "3cc43103-ea9a-42c2-99c4-f8688d5136b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:52:13.767281 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:13.767206    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc43103-ea9a-42c2-99c4-f8688d5136b1-kube-api-access-shggw" (OuterVolumeSpecName: "kube-api-access-shggw") pod "3cc43103-ea9a-42c2-99c4-f8688d5136b1" (UID: "3cc43103-ea9a-42c2-99c4-f8688d5136b1"). InnerVolumeSpecName "kube-api-access-shggw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:52:13.767281 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:13.767244    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cc43103-ea9a-42c2-99c4-f8688d5136b1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3cc43103-ea9a-42c2-99c4-f8688d5136b1" (UID: "3cc43103-ea9a-42c2-99c4-f8688d5136b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:52:13.865974 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:13.865937    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc43103-ea9a-42c2-99c4-f8688d5136b1-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:13.865974 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:13.865969    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc43103-ea9a-42c2-99c4-f8688d5136b1-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:13.865974 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:13.865978    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc43103-ea9a-42c2-99c4-f8688d5136b1-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:13.866219 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:13.865988    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shggw\" (UniqueName: \"kubernetes.io/projected/3cc43103-ea9a-42c2-99c4-f8688d5136b1-kube-api-access-shggw\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:14.239090 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:14.239053    2562 generic.go:356] "Generic (PLEG): container finished" podID="3cc43103-ea9a-42c2-99c4-f8688d5136b1" containerID="a7a2ce9793b48cd07cdd8e2a24ceba9cb70144723919d8163400d865160879ab" exitCode=0
Sep 04 23:52:14.239265 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:14.239094    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"
Sep 04 23:52:14.239265 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:14.239119    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz" event={"ID":"3cc43103-ea9a-42c2-99c4-f8688d5136b1","Type":"ContainerDied","Data":"a7a2ce9793b48cd07cdd8e2a24ceba9cb70144723919d8163400d865160879ab"}
Sep 04 23:52:14.239265 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:14.239168    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz" event={"ID":"3cc43103-ea9a-42c2-99c4-f8688d5136b1","Type":"ContainerDied","Data":"797267716f841140a8a676681bba5adc4f603ce4d8e7bc5ab913d4298241d0b2"}
Sep 04 23:52:14.239265 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:14.239192    2562 scope.go:117] "RemoveContainer" containerID="a7a2ce9793b48cd07cdd8e2a24ceba9cb70144723919d8163400d865160879ab"
Sep 04 23:52:14.249229 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:14.249204    2562 scope.go:117] "RemoveContainer" containerID="a7a2ce9793b48cd07cdd8e2a24ceba9cb70144723919d8163400d865160879ab"
Sep 04 23:52:14.249514 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:52:14.249496    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a2ce9793b48cd07cdd8e2a24ceba9cb70144723919d8163400d865160879ab\": container with ID starting with a7a2ce9793b48cd07cdd8e2a24ceba9cb70144723919d8163400d865160879ab not found: ID does not exist" containerID="a7a2ce9793b48cd07cdd8e2a24ceba9cb70144723919d8163400d865160879ab"
Sep 04 23:52:14.249567 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:14.249523    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a2ce9793b48cd07cdd8e2a24ceba9cb70144723919d8163400d865160879ab"} err="failed to get container status \"a7a2ce9793b48cd07cdd8e2a24ceba9cb70144723919d8163400d865160879ab\": rpc error: code = NotFound desc = could not find container \"a7a2ce9793b48cd07cdd8e2a24ceba9cb70144723919d8163400d865160879ab\": container with ID starting with a7a2ce9793b48cd07cdd8e2a24ceba9cb70144723919d8163400d865160879ab not found: ID does not exist"
Sep 04 23:52:14.263478 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:14.263452    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"]
Sep 04 23:52:14.265302 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:14.265281    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6fa8d6-854c6d6d8f-6qwqz"]
Sep 04 23:52:14.358433 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:14.358398    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cc43103-ea9a-42c2-99c4-f8688d5136b1" path="/var/lib/kubelet/pods/3cc43103-ea9a-42c2-99c4-f8688d5136b1/volumes"
Sep 04 23:52:20.756958 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.756922    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"]
Sep 04 23:52:20.757372 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.757358    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerName="istio-proxy"
Sep 04 23:52:20.757416 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.757374    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerName="istio-proxy"
Sep 04 23:52:20.757416 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.757383    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cc43103-ea9a-42c2-99c4-f8688d5136b1" containerName="maas-api"
Sep 04 23:52:20.757416 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.757390    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cc43103-ea9a-42c2-99c4-f8688d5136b1" containerName="maas-api"
Sep 04 23:52:20.757416 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.757412    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerName="istio-proxy"
Sep 04 23:52:20.757538 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.757418    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerName="istio-proxy"
Sep 04 23:52:20.757538 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.757497    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerName="istio-proxy"
Sep 04 23:52:20.757538 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.757507    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="976dc20d-a7b7-4894-bfd6-d2c84141c97e" containerName="istio-proxy"
Sep 04 23:52:20.757538 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.757514    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cc43103-ea9a-42c2-99c4-f8688d5136b1" containerName="maas-api"
Sep 04 23:52:20.762027 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.762007    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.764583 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.764560    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-ffeb50-openshift-default-dockercfg-mtxjg\""
Sep 04 23:52:20.773340 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.773315    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"]
Sep 04 23:52:20.831341 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.831303    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-workload-certs\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.831541 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.831387    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-credential-socket\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.831541 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.831420    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnssr\" (UniqueName: \"kubernetes.io/projected/29690154-4057-4e6c-8d01-57b9aaf1964b-kube-api-access-rnssr\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.831541 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.831447    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29690154-4057-4e6c-8d01-57b9aaf1964b-istiod-ca-cert\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.831541 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.831525    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-workload-socket\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.831754 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.831590    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-envoy\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.831754 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.831622    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-token\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.831754 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.831663    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-podinfo\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.831754 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.831699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-data\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.932458 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.932415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-envoy\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.932458 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.932461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-token\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.932760 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.932487    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-podinfo\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.932760 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.932518    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-data\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.932760 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.932584    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-workload-certs\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.932760 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.932639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-credential-socket\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.932760 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.932696    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnssr\" (UniqueName: \"kubernetes.io/projected/29690154-4057-4e6c-8d01-57b9aaf1964b-kube-api-access-rnssr\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.932760 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.932725    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29690154-4057-4e6c-8d01-57b9aaf1964b-istiod-ca-cert\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.933087 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.932768    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-workload-socket\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.933087 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.932968    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-data\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.933087 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.933050    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-workload-certs\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.933233 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.933118    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-workload-socket\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.933233 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.933208    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-credential-socket\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.933483 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.933464    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29690154-4057-4e6c-8d01-57b9aaf1964b-istiod-ca-cert\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.934737 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.934715    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-envoy\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.934947 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.934928    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-podinfo\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.939975 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.939943    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-token\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:20.940080 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:20.940003    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnssr\" (UniqueName: \"kubernetes.io/projected/29690154-4057-4e6c-8d01-57b9aaf1964b-kube-api-access-rnssr\") pod \"e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") " pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:21.073637 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:21.073538    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:21.347751 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:21.347718    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"]
Sep 04 23:52:21.349766 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:52:21.349733    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29690154_4057_4e6c_8d01_57b9aaf1964b.slice/crio-b0cd037698d2e128bc0c4ce610062dfe1d9f54fe1dc039c9ba4dcb73be9a04ad WatchSource:0}: Error finding container b0cd037698d2e128bc0c4ce610062dfe1d9f54fe1dc039c9ba4dcb73be9a04ad: Status 404 returned error can't find the container with id b0cd037698d2e128bc0c4ce610062dfe1d9f54fe1dc039c9ba4dcb73be9a04ad
Sep 04 23:52:21.351949 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:21.351913    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:52:21.352068 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:21.351984    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:52:21.352068 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:21.352013    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:52:22.292291 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:22.292247    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx" event={"ID":"29690154-4057-4e6c-8d01-57b9aaf1964b","Type":"ContainerStarted","Data":"4f435810fdbeb49a55aa26154b7737a0b2a413cae5dbcb772f80a42d2a9a709b"}
Sep 04 23:52:22.292698 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:22.292355    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx" event={"ID":"29690154-4057-4e6c-8d01-57b9aaf1964b","Type":"ContainerStarted","Data":"b0cd037698d2e128bc0c4ce610062dfe1d9f54fe1dc039c9ba4dcb73be9a04ad"}
Sep 04 23:52:22.313106 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:22.313060    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx" podStartSLOduration=2.313043296 podStartE2EDuration="2.313043296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:52:22.310961987 +0000 UTC m=+954.557629066" watchObservedRunningTime="2026-09-04 23:52:22.313043296 +0000 UTC m=+954.559710418"
Sep 04 23:52:23.074448 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:23.074413    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:23.079208 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:23.079183    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:23.297797 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:23.297762    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:23.298799 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:23.298778    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:35.167884 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.167850    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"]
Sep 04 23:52:35.173893 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.173868    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:35.176273 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.176251    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f4c99d38-serving-cert\""
Sep 04 23:52:35.185868 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.185814    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"]
Sep 04 23:52:35.271565 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.271508    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt5ks\" (UniqueName: \"kubernetes.io/projected/180491bd-e5f4-4ed6-850b-a20d8dde5695-kube-api-access-zt5ks\") pod \"maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:35.271763 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.271578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/180491bd-e5f4-4ed6-850b-a20d8dde5695-metrics-certs\") pod \"maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:35.271763 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.271697    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/180491bd-e5f4-4ed6-850b-a20d8dde5695-openshift-service-ca\") pod \"maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:35.271763 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.271732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/180491bd-e5f4-4ed6-850b-a20d8dde5695-maas-api-tls\") pod \"maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:35.372884 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.372849    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/180491bd-e5f4-4ed6-850b-a20d8dde5695-openshift-service-ca\") pod \"maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:35.372884 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.372901    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/180491bd-e5f4-4ed6-850b-a20d8dde5695-maas-api-tls\") pod \"maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:35.373263 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.373235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt5ks\" (UniqueName: \"kubernetes.io/projected/180491bd-e5f4-4ed6-850b-a20d8dde5695-kube-api-access-zt5ks\") pod \"maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:35.373566 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.373540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/180491bd-e5f4-4ed6-850b-a20d8dde5695-metrics-certs\") pod \"maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:35.374709 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.374684    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/180491bd-e5f4-4ed6-850b-a20d8dde5695-openshift-service-ca\") pod \"maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:35.376260 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.376241    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/180491bd-e5f4-4ed6-850b-a20d8dde5695-maas-api-tls\") pod \"maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:35.376364 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.376344    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/180491bd-e5f4-4ed6-850b-a20d8dde5695-metrics-certs\") pod \"maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:35.380436 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.380417    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt5ks\" (UniqueName: \"kubernetes.io/projected/180491bd-e5f4-4ed6-850b-a20d8dde5695-kube-api-access-zt5ks\") pod \"maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:35.487463 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.487420    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:35.724970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.724928    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"]
Sep 04 23:52:35.725194 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.725169    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx" podUID="29690154-4057-4e6c-8d01-57b9aaf1964b" containerName="istio-proxy" containerID="cri-o://4f435810fdbeb49a55aa26154b7737a0b2a413cae5dbcb772f80a42d2a9a709b" gracePeriod=30
Sep 04 23:52:35.765347 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.765322    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"]
Sep 04 23:52:35.767490 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:52:35.767462    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180491bd_e5f4_4ed6_850b_a20d8dde5695.slice/crio-d467ca336c500f7d4cc0072c443c64e597640266ea20ec141df43cfdb8cabbba WatchSource:0}: Error finding container d467ca336c500f7d4cc0072c443c64e597640266ea20ec141df43cfdb8cabbba: Status 404 returned error can't find the container with id d467ca336c500f7d4cc0072c443c64e597640266ea20ec141df43cfdb8cabbba
Sep 04 23:52:35.769049 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:35.769030    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 23:52:36.368244 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:36.368204    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44" event={"ID":"180491bd-e5f4-4ed6-850b-a20d8dde5695","Type":"ContainerStarted","Data":"d467ca336c500f7d4cc0072c443c64e597640266ea20ec141df43cfdb8cabbba"}
Sep 04 23:52:37.375754 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:37.375633    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44" event={"ID":"180491bd-e5f4-4ed6-850b-a20d8dde5695","Type":"ContainerStarted","Data":"dc3827cbe60126b1861248a10bd8def727a58917c05b54337b19e8ccc493f4d1"}
Sep 04 23:52:37.398193 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:37.398139    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44" podStartSLOduration=1.112305422 podStartE2EDuration="2.398120522s" totalImagesPullingTime="1.2858151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:52:37.395330498 +0000 UTC m=+969.641997578" watchObservedRunningTime="2026-09-04 23:52:37.398120522 +0000 UTC m=+969.644787601"
Sep 04 23:52:38.300102 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:38.300065    2562 patch_prober.go:28] interesting pod/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 04 23:52:38.300304 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:38.300121    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx" podUID="29690154-4057-4e6c-8d01-57b9aaf1964b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 23:52:38.380269 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:38.380237    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:40.983296 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:40.983274    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:41.130446 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.130346    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-data\") pod \"29690154-4057-4e6c-8d01-57b9aaf1964b\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") "
Sep 04 23:52:41.130446 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.130417    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-token\") pod \"29690154-4057-4e6c-8d01-57b9aaf1964b\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") "
Sep 04 23:52:41.130712 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.130481    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-workload-certs\") pod \"29690154-4057-4e6c-8d01-57b9aaf1964b\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") "
Sep 04 23:52:41.130712 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.130505    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnssr\" (UniqueName: \"kubernetes.io/projected/29690154-4057-4e6c-8d01-57b9aaf1964b-kube-api-access-rnssr\") pod \"29690154-4057-4e6c-8d01-57b9aaf1964b\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") "
Sep 04 23:52:41.130712 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.130584    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-credential-socket\") pod \"29690154-4057-4e6c-8d01-57b9aaf1964b\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") "
Sep 04 23:52:41.130712 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.130611    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-podinfo\") pod \"29690154-4057-4e6c-8d01-57b9aaf1964b\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") "
Sep 04 23:52:41.130712 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.130645    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29690154-4057-4e6c-8d01-57b9aaf1964b-istiod-ca-cert\") pod \"29690154-4057-4e6c-8d01-57b9aaf1964b\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") "
Sep 04 23:52:41.130712 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.130694    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-envoy\") pod \"29690154-4057-4e6c-8d01-57b9aaf1964b\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") "
Sep 04 23:52:41.131017 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.130703    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-data" (OuterVolumeSpecName: "istio-data") pod "29690154-4057-4e6c-8d01-57b9aaf1964b" (UID: "29690154-4057-4e6c-8d01-57b9aaf1964b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:41.131017 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.130735    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "29690154-4057-4e6c-8d01-57b9aaf1964b" (UID: "29690154-4057-4e6c-8d01-57b9aaf1964b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:41.131017 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.130740    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-workload-socket\") pod \"29690154-4057-4e6c-8d01-57b9aaf1964b\" (UID: \"29690154-4057-4e6c-8d01-57b9aaf1964b\") "
Sep 04 23:52:41.131017 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.130876    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "29690154-4057-4e6c-8d01-57b9aaf1964b" (UID: "29690154-4057-4e6c-8d01-57b9aaf1964b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:41.131017 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.130937    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "29690154-4057-4e6c-8d01-57b9aaf1964b" (UID: "29690154-4057-4e6c-8d01-57b9aaf1964b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:41.131234 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.131046    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29690154-4057-4e6c-8d01-57b9aaf1964b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "29690154-4057-4e6c-8d01-57b9aaf1964b" (UID: "29690154-4057-4e6c-8d01-57b9aaf1964b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:52:41.131433 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.131408    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-credential-socket\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:41.131541 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.131440    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29690154-4057-4e6c-8d01-57b9aaf1964b-istiod-ca-cert\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:41.131541 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.131455    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-workload-socket\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:41.131541 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.131468    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-data\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:41.131541 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.131482    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-workload-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:41.133095 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.133057    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29690154-4057-4e6c-8d01-57b9aaf1964b-kube-api-access-rnssr" (OuterVolumeSpecName: "kube-api-access-rnssr") pod "29690154-4057-4e6c-8d01-57b9aaf1964b" (UID: "29690154-4057-4e6c-8d01-57b9aaf1964b"). InnerVolumeSpecName "kube-api-access-rnssr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:52:41.133217 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.133157    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "29690154-4057-4e6c-8d01-57b9aaf1964b" (UID: "29690154-4057-4e6c-8d01-57b9aaf1964b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:52:41.133327 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.133298    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-token" (OuterVolumeSpecName: "istio-token") pod "29690154-4057-4e6c-8d01-57b9aaf1964b" (UID: "29690154-4057-4e6c-8d01-57b9aaf1964b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:52:41.133373 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.133314    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "29690154-4057-4e6c-8d01-57b9aaf1964b" (UID: "29690154-4057-4e6c-8d01-57b9aaf1964b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:41.232642 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.232605    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-token\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:41.232642 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.232635    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnssr\" (UniqueName: \"kubernetes.io/projected/29690154-4057-4e6c-8d01-57b9aaf1964b-kube-api-access-rnssr\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:41.232642 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.232647    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-podinfo\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:41.232913 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.232685    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29690154-4057-4e6c-8d01-57b9aaf1964b-istio-envoy\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:41.397586 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.397495    2562 generic.go:356] "Generic (PLEG): container finished" podID="29690154-4057-4e6c-8d01-57b9aaf1964b" containerID="4f435810fdbeb49a55aa26154b7737a0b2a413cae5dbcb772f80a42d2a9a709b" exitCode=0
Sep 04 23:52:41.397586 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.397541    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"
Sep 04 23:52:41.397586 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.397573    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx" event={"ID":"29690154-4057-4e6c-8d01-57b9aaf1964b","Type":"ContainerDied","Data":"4f435810fdbeb49a55aa26154b7737a0b2a413cae5dbcb772f80a42d2a9a709b"}
Sep 04 23:52:41.397854 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.397627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx" event={"ID":"29690154-4057-4e6c-8d01-57b9aaf1964b","Type":"ContainerDied","Data":"b0cd037698d2e128bc0c4ce610062dfe1d9f54fe1dc039c9ba4dcb73be9a04ad"}
Sep 04 23:52:41.397854 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.397674    2562 scope.go:117] "RemoveContainer" containerID="4f435810fdbeb49a55aa26154b7737a0b2a413cae5dbcb772f80a42d2a9a709b"
Sep 04 23:52:41.407578 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.407560    2562 scope.go:117] "RemoveContainer" containerID="4f435810fdbeb49a55aa26154b7737a0b2a413cae5dbcb772f80a42d2a9a709b"
Sep 04 23:52:41.407896 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:52:41.407874    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f435810fdbeb49a55aa26154b7737a0b2a413cae5dbcb772f80a42d2a9a709b\": container with ID starting with 4f435810fdbeb49a55aa26154b7737a0b2a413cae5dbcb772f80a42d2a9a709b not found: ID does not exist" containerID="4f435810fdbeb49a55aa26154b7737a0b2a413cae5dbcb772f80a42d2a9a709b"
Sep 04 23:52:41.407965 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.407907    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f435810fdbeb49a55aa26154b7737a0b2a413cae5dbcb772f80a42d2a9a709b"} err="failed to get container status \"4f435810fdbeb49a55aa26154b7737a0b2a413cae5dbcb772f80a42d2a9a709b\": rpc error: code = NotFound desc = could not find container \"4f435810fdbeb49a55aa26154b7737a0b2a413cae5dbcb772f80a42d2a9a709b\": container with ID starting with 4f435810fdbeb49a55aa26154b7737a0b2a413cae5dbcb772f80a42d2a9a709b not found: ID does not exist"
Sep 04 23:52:41.419135 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.419112    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"]
Sep 04 23:52:41.425342 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:41.425320    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-ffeb50-openshift-default-6fc657db8d-hsxmx"]
Sep 04 23:52:42.358723 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:42.358692    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29690154-4057-4e6c-8d01-57b9aaf1964b" path="/var/lib/kubelet/pods/29690154-4057-4e6c-8d01-57b9aaf1964b/volumes"
Sep 04 23:52:44.391110 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:44.391080    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:52:50.728178 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:50.728141    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"]
Sep 04 23:52:50.728680 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:50.728414    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb" podUID="76b631a9-360b-4eab-a373-d420d2fba2a9" containerName="maas-api" containerID="cri-o://be820bb5877e88b5a45606ece7b2c50a88afd686d69e56d1775b281db67d7dee" gracePeriod=30
Sep 04 23:52:50.990425 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:50.990363    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:52:51.017093 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.017061    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5vl7\" (UniqueName: \"kubernetes.io/projected/76b631a9-360b-4eab-a373-d420d2fba2a9-kube-api-access-k5vl7\") pod \"76b631a9-360b-4eab-a373-d420d2fba2a9\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") "
Sep 04 23:52:51.017254 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.017152    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b631a9-360b-4eab-a373-d420d2fba2a9-metrics-certs\") pod \"76b631a9-360b-4eab-a373-d420d2fba2a9\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") "
Sep 04 23:52:51.017254 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.017201    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b631a9-360b-4eab-a373-d420d2fba2a9-openshift-service-ca\") pod \"76b631a9-360b-4eab-a373-d420d2fba2a9\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") "
Sep 04 23:52:51.017552 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.017523    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b631a9-360b-4eab-a373-d420d2fba2a9-maas-api-tls\") pod \"76b631a9-360b-4eab-a373-d420d2fba2a9\" (UID: \"76b631a9-360b-4eab-a373-d420d2fba2a9\") "
Sep 04 23:52:51.017734 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.017527    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b631a9-360b-4eab-a373-d420d2fba2a9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "76b631a9-360b-4eab-a373-d420d2fba2a9" (UID: "76b631a9-360b-4eab-a373-d420d2fba2a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:52:51.017913 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.017886    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b631a9-360b-4eab-a373-d420d2fba2a9-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:51.019407 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.019382    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b631a9-360b-4eab-a373-d420d2fba2a9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "76b631a9-360b-4eab-a373-d420d2fba2a9" (UID: "76b631a9-360b-4eab-a373-d420d2fba2a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:52:51.019519 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.019505    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b631a9-360b-4eab-a373-d420d2fba2a9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "76b631a9-360b-4eab-a373-d420d2fba2a9" (UID: "76b631a9-360b-4eab-a373-d420d2fba2a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:52:51.019707 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.019691    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b631a9-360b-4eab-a373-d420d2fba2a9-kube-api-access-k5vl7" (OuterVolumeSpecName: "kube-api-access-k5vl7") pod "76b631a9-360b-4eab-a373-d420d2fba2a9" (UID: "76b631a9-360b-4eab-a373-d420d2fba2a9"). InnerVolumeSpecName "kube-api-access-k5vl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:52:51.119032 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.118995    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5vl7\" (UniqueName: \"kubernetes.io/projected/76b631a9-360b-4eab-a373-d420d2fba2a9-kube-api-access-k5vl7\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:51.119032 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.119025    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b631a9-360b-4eab-a373-d420d2fba2a9-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:51.119032 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.119034    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b631a9-360b-4eab-a373-d420d2fba2a9-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:52:51.451429 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.451331    2562 generic.go:356] "Generic (PLEG): container finished" podID="76b631a9-360b-4eab-a373-d420d2fba2a9" containerID="be820bb5877e88b5a45606ece7b2c50a88afd686d69e56d1775b281db67d7dee" exitCode=0
Sep 04 23:52:51.451429 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.451384    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"
Sep 04 23:52:51.451429 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.451418    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb" event={"ID":"76b631a9-360b-4eab-a373-d420d2fba2a9","Type":"ContainerDied","Data":"be820bb5877e88b5a45606ece7b2c50a88afd686d69e56d1775b281db67d7dee"}
Sep 04 23:52:51.451750 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.451473    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb" event={"ID":"76b631a9-360b-4eab-a373-d420d2fba2a9","Type":"ContainerDied","Data":"147aceb1b0a82ccb7efe74fccaeac6a4af51619550192c431b9f25691c5152b7"}
Sep 04 23:52:51.451750 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.451494    2562 scope.go:117] "RemoveContainer" containerID="be820bb5877e88b5a45606ece7b2c50a88afd686d69e56d1775b281db67d7dee"
Sep 04 23:52:51.461561 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.461544    2562 scope.go:117] "RemoveContainer" containerID="be820bb5877e88b5a45606ece7b2c50a88afd686d69e56d1775b281db67d7dee"
Sep 04 23:52:51.461889 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:52:51.461870    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be820bb5877e88b5a45606ece7b2c50a88afd686d69e56d1775b281db67d7dee\": container with ID starting with be820bb5877e88b5a45606ece7b2c50a88afd686d69e56d1775b281db67d7dee not found: ID does not exist" containerID="be820bb5877e88b5a45606ece7b2c50a88afd686d69e56d1775b281db67d7dee"
Sep 04 23:52:51.461970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.461897    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be820bb5877e88b5a45606ece7b2c50a88afd686d69e56d1775b281db67d7dee"} err="failed to get container status \"be820bb5877e88b5a45606ece7b2c50a88afd686d69e56d1775b281db67d7dee\": rpc error: code = NotFound desc = could not find container \"be820bb5877e88b5a45606ece7b2c50a88afd686d69e56d1775b281db67d7dee\": container with ID starting with be820bb5877e88b5a45606ece7b2c50a88afd686d69e56d1775b281db67d7dee not found: ID does not exist"
Sep 04 23:52:51.472707 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.472682    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"]
Sep 04 23:52:51.476125 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:51.476105    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d8b443-569dd66758-dqfkb"]
Sep 04 23:52:52.359567 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:52:52.359531    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76b631a9-360b-4eab-a373-d420d2fba2a9" path="/var/lib/kubelet/pods/76b631a9-360b-4eab-a373-d420d2fba2a9/volumes"
Sep 04 23:53:08.493167 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:08.493131    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"]
Sep 04 23:53:08.493817 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:08.493394    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn" podUID="0677400f-dc86-4217-b126-b807feeabecd" containerName="maas-api" containerID="cri-o://1df92991b6efde54a0b0dbfdeba780ae31ee4cad4c35f3b68327889ffa50f414" gracePeriod=30
Sep 04 23:53:08.752810 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:08.752745    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:53:08.891494 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:08.891460    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0677400f-dc86-4217-b126-b807feeabecd-maas-api-tls\") pod \"0677400f-dc86-4217-b126-b807feeabecd\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") "
Sep 04 23:53:08.891685 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:08.891542    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0677400f-dc86-4217-b126-b807feeabecd-openshift-service-ca\") pod \"0677400f-dc86-4217-b126-b807feeabecd\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") "
Sep 04 23:53:08.891685 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:08.891593    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mdzb\" (UniqueName: \"kubernetes.io/projected/0677400f-dc86-4217-b126-b807feeabecd-kube-api-access-2mdzb\") pod \"0677400f-dc86-4217-b126-b807feeabecd\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") "
Sep 04 23:53:08.891685 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:08.891639    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0677400f-dc86-4217-b126-b807feeabecd-metrics-certs\") pod \"0677400f-dc86-4217-b126-b807feeabecd\" (UID: \"0677400f-dc86-4217-b126-b807feeabecd\") "
Sep 04 23:53:08.891942 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:08.891917    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0677400f-dc86-4217-b126-b807feeabecd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0677400f-dc86-4217-b126-b807feeabecd" (UID: "0677400f-dc86-4217-b126-b807feeabecd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:53:08.893664 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:08.893623    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0677400f-dc86-4217-b126-b807feeabecd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0677400f-dc86-4217-b126-b807feeabecd" (UID: "0677400f-dc86-4217-b126-b807feeabecd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:53:08.893791 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:08.893646    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0677400f-dc86-4217-b126-b807feeabecd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0677400f-dc86-4217-b126-b807feeabecd" (UID: "0677400f-dc86-4217-b126-b807feeabecd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:53:08.893791 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:08.893722    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0677400f-dc86-4217-b126-b807feeabecd-kube-api-access-2mdzb" (OuterVolumeSpecName: "kube-api-access-2mdzb") pod "0677400f-dc86-4217-b126-b807feeabecd" (UID: "0677400f-dc86-4217-b126-b807feeabecd"). InnerVolumeSpecName "kube-api-access-2mdzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:53:08.993249 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:08.993220    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0677400f-dc86-4217-b126-b807feeabecd-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:53:08.993249 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:08.993244    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0677400f-dc86-4217-b126-b807feeabecd-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:53:08.993249 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:08.993254    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0677400f-dc86-4217-b126-b807feeabecd-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:53:08.993465 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:08.993263    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mdzb\" (UniqueName: \"kubernetes.io/projected/0677400f-dc86-4217-b126-b807feeabecd-kube-api-access-2mdzb\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:53:09.557349 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:09.557318    2562 generic.go:356] "Generic (PLEG): container finished" podID="0677400f-dc86-4217-b126-b807feeabecd" containerID="1df92991b6efde54a0b0dbfdeba780ae31ee4cad4c35f3b68327889ffa50f414" exitCode=0
Sep 04 23:53:09.557803 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:09.557363    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"
Sep 04 23:53:09.557803 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:09.557408    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn" event={"ID":"0677400f-dc86-4217-b126-b807feeabecd","Type":"ContainerDied","Data":"1df92991b6efde54a0b0dbfdeba780ae31ee4cad4c35f3b68327889ffa50f414"}
Sep 04 23:53:09.557803 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:09.557459    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn" event={"ID":"0677400f-dc86-4217-b126-b807feeabecd","Type":"ContainerDied","Data":"784e05dad0dffa7a7f2b67deca85461ee97d51fed08f1a90b0980844850f23a6"}
Sep 04 23:53:09.557803 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:09.557482    2562 scope.go:117] "RemoveContainer" containerID="1df92991b6efde54a0b0dbfdeba780ae31ee4cad4c35f3b68327889ffa50f414"
Sep 04 23:53:09.567545 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:09.567525    2562 scope.go:117] "RemoveContainer" containerID="1df92991b6efde54a0b0dbfdeba780ae31ee4cad4c35f3b68327889ffa50f414"
Sep 04 23:53:09.567839 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:53:09.567819    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df92991b6efde54a0b0dbfdeba780ae31ee4cad4c35f3b68327889ffa50f414\": container with ID starting with 1df92991b6efde54a0b0dbfdeba780ae31ee4cad4c35f3b68327889ffa50f414 not found: ID does not exist" containerID="1df92991b6efde54a0b0dbfdeba780ae31ee4cad4c35f3b68327889ffa50f414"
Sep 04 23:53:09.567896 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:09.567849    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df92991b6efde54a0b0dbfdeba780ae31ee4cad4c35f3b68327889ffa50f414"} err="failed to get container status \"1df92991b6efde54a0b0dbfdeba780ae31ee4cad4c35f3b68327889ffa50f414\": rpc error: code = NotFound desc = could not find container \"1df92991b6efde54a0b0dbfdeba780ae31ee4cad4c35f3b68327889ffa50f414\": container with ID starting with 1df92991b6efde54a0b0dbfdeba780ae31ee4cad4c35f3b68327889ffa50f414 not found: ID does not exist"
Sep 04 23:53:09.578031 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:09.578005    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"]
Sep 04 23:53:09.582090 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:09.582069    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-724e1f-fbdd4bc57-gszpn"]
Sep 04 23:53:10.357888 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:10.357856    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0677400f-dc86-4217-b126-b807feeabecd" path="/var/lib/kubelet/pods/0677400f-dc86-4217-b126-b807feeabecd/volumes"
Sep 04 23:53:15.730738 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.730706    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"]
Sep 04 23:53:15.731221 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.731201    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b631a9-360b-4eab-a373-d420d2fba2a9" containerName="maas-api"
Sep 04 23:53:15.731221 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.731220    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b631a9-360b-4eab-a373-d420d2fba2a9" containerName="maas-api"
Sep 04 23:53:15.731341 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.731228    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29690154-4057-4e6c-8d01-57b9aaf1964b" containerName="istio-proxy"
Sep 04 23:53:15.731341 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.731233    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="29690154-4057-4e6c-8d01-57b9aaf1964b" containerName="istio-proxy"
Sep 04 23:53:15.731341 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.731253    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0677400f-dc86-4217-b126-b807feeabecd" containerName="maas-api"
Sep 04 23:53:15.731341 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.731258    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0677400f-dc86-4217-b126-b807feeabecd" containerName="maas-api"
Sep 04 23:53:15.731341 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.731328    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="29690154-4057-4e6c-8d01-57b9aaf1964b" containerName="istio-proxy"
Sep 04 23:53:15.731341 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.731338    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b631a9-360b-4eab-a373-d420d2fba2a9" containerName="maas-api"
Sep 04 23:53:15.731629 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.731345    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0677400f-dc86-4217-b126-b807feeabecd" containerName="maas-api"
Sep 04 23:53:15.736032 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.736009    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.738365 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.738344    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-5245c1-openshift-default-dockercfg-f5dd7\""
Sep 04 23:53:15.747896 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.747874    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"]
Sep 04 23:53:15.856527 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.856490    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-credential-socket\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.856764 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.856543    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-token\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.856764 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.856624    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istiod-ca-cert\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.856764 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.856686    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-podinfo\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.856764 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.856705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-workload-certs\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.856993 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.856777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-envoy\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.856993 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.856841    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-workload-socket\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.856993 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.856881    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvcdk\" (UniqueName: \"kubernetes.io/projected/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-kube-api-access-kvcdk\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.856993 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.856923    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-data\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.957938 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.957901    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-data\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.958115 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.957946    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-credential-socket\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.958115 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.957990    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-token\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.958115 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.958025    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istiod-ca-cert\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.958115 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.958061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-podinfo\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.958115 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.958087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-workload-certs\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.958377 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.958126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-envoy\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.958377 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.958344    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-workload-socket\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.958485 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.958367    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-data\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.958485 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.958389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvcdk\" (UniqueName: \"kubernetes.io/projected/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-kube-api-access-kvcdk\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.958485 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.958428    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-credential-socket\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.958721 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.958702    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istiod-ca-cert\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.958861 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.958842    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-workload-socket\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.959018 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.958994    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-workload-certs\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.960433 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.960406    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-envoy\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.960970 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.960948    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-podinfo\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.965862 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.965834    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-token\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:15.965997 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:15.965980    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvcdk\" (UniqueName: \"kubernetes.io/projected/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-kube-api-access-kvcdk\") pod \"e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") " pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:16.049665 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.049562    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:16.327468 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.327434    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"]
Sep 04 23:53:16.329940 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:53:16.329905    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca9520e_1f61_4e0d_bdd8_19f6cfb857de.slice/crio-a135e65e0e761922a7694c9271280c55f3a5a2cde666bb2b9ad56a415060769c WatchSource:0}: Error finding container a135e65e0e761922a7694c9271280c55f3a5a2cde666bb2b9ad56a415060769c: Status 404 returned error can't find the container with id a135e65e0e761922a7694c9271280c55f3a5a2cde666bb2b9ad56a415060769c
Sep 04 23:53:16.332723 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.332688    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:53:16.332821 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.332755    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:53:16.332821 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.332785    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:53:16.598967 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.598861    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf" event={"ID":"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de","Type":"ContainerStarted","Data":"26ba251087b4b397ad44c4de837187ac8b0fc24e52542773c289e0be4c869045"}
Sep 04 23:53:16.598967 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.598935    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf" event={"ID":"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de","Type":"ContainerStarted","Data":"a135e65e0e761922a7694c9271280c55f3a5a2cde666bb2b9ad56a415060769c"}
Sep 04 23:53:16.619468 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.619326    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf" podStartSLOduration=1.6193034370000001 podStartE2EDuration="1.619303437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:53:16.618443579 +0000 UTC m=+1008.865110658" watchObservedRunningTime="2026-09-04 23:53:16.619303437 +0000 UTC m=+1008.865970517"
Sep 04 23:53:16.626222 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.626180    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"]
Sep 04 23:53:16.626446 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.626420    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44" podUID="180491bd-e5f4-4ed6-850b-a20d8dde5695" containerName="maas-api" containerID="cri-o://dc3827cbe60126b1861248a10bd8def727a58917c05b54337b19e8ccc493f4d1" gracePeriod=30
Sep 04 23:53:16.865897 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.865871    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:53:16.969272 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.969235    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/180491bd-e5f4-4ed6-850b-a20d8dde5695-metrics-certs\") pod \"180491bd-e5f4-4ed6-850b-a20d8dde5695\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") "
Sep 04 23:53:16.969403 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.969378    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/180491bd-e5f4-4ed6-850b-a20d8dde5695-maas-api-tls\") pod \"180491bd-e5f4-4ed6-850b-a20d8dde5695\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") "
Sep 04 23:53:16.969472 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.969426    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zt5ks\" (UniqueName: \"kubernetes.io/projected/180491bd-e5f4-4ed6-850b-a20d8dde5695-kube-api-access-zt5ks\") pod \"180491bd-e5f4-4ed6-850b-a20d8dde5695\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") "
Sep 04 23:53:16.969529 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.969485    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/180491bd-e5f4-4ed6-850b-a20d8dde5695-openshift-service-ca\") pod \"180491bd-e5f4-4ed6-850b-a20d8dde5695\" (UID: \"180491bd-e5f4-4ed6-850b-a20d8dde5695\") "
Sep 04 23:53:16.969972 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.969940    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180491bd-e5f4-4ed6-850b-a20d8dde5695-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "180491bd-e5f4-4ed6-850b-a20d8dde5695" (UID: "180491bd-e5f4-4ed6-850b-a20d8dde5695"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:53:16.971413 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.971385    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/180491bd-e5f4-4ed6-850b-a20d8dde5695-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "180491bd-e5f4-4ed6-850b-a20d8dde5695" (UID: "180491bd-e5f4-4ed6-850b-a20d8dde5695"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:53:16.971704 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.971678    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/180491bd-e5f4-4ed6-850b-a20d8dde5695-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "180491bd-e5f4-4ed6-850b-a20d8dde5695" (UID: "180491bd-e5f4-4ed6-850b-a20d8dde5695"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:53:16.971897 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:16.971761    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180491bd-e5f4-4ed6-850b-a20d8dde5695-kube-api-access-zt5ks" (OuterVolumeSpecName: "kube-api-access-zt5ks") pod "180491bd-e5f4-4ed6-850b-a20d8dde5695" (UID: "180491bd-e5f4-4ed6-850b-a20d8dde5695"). InnerVolumeSpecName "kube-api-access-zt5ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:53:17.050320 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.050283    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:17.055398 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.055372    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:17.070764 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.070741    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/180491bd-e5f4-4ed6-850b-a20d8dde5695-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:53:17.070764 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.070762    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt5ks\" (UniqueName: \"kubernetes.io/projected/180491bd-e5f4-4ed6-850b-a20d8dde5695-kube-api-access-zt5ks\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:53:17.070953 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.070771    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/180491bd-e5f4-4ed6-850b-a20d8dde5695-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:53:17.070953 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.070781    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/180491bd-e5f4-4ed6-850b-a20d8dde5695-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:53:17.604815 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.604783    2562 generic.go:356] "Generic (PLEG): container finished" podID="180491bd-e5f4-4ed6-850b-a20d8dde5695" containerID="dc3827cbe60126b1861248a10bd8def727a58917c05b54337b19e8ccc493f4d1" exitCode=0
Sep 04 23:53:17.604986 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.604842    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"
Sep 04 23:53:17.604986 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.604867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44" event={"ID":"180491bd-e5f4-4ed6-850b-a20d8dde5695","Type":"ContainerDied","Data":"dc3827cbe60126b1861248a10bd8def727a58917c05b54337b19e8ccc493f4d1"}
Sep 04 23:53:17.604986 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.604916    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44" event={"ID":"180491bd-e5f4-4ed6-850b-a20d8dde5695","Type":"ContainerDied","Data":"d467ca336c500f7d4cc0072c443c64e597640266ea20ec141df43cfdb8cabbba"}
Sep 04 23:53:17.604986 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.604938    2562 scope.go:117] "RemoveContainer" containerID="dc3827cbe60126b1861248a10bd8def727a58917c05b54337b19e8ccc493f4d1"
Sep 04 23:53:17.605394 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.605338    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:17.606321 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.606289    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:17.614109 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.614093    2562 scope.go:117] "RemoveContainer" containerID="dc3827cbe60126b1861248a10bd8def727a58917c05b54337b19e8ccc493f4d1"
Sep 04 23:53:17.614383 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:53:17.614361    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc3827cbe60126b1861248a10bd8def727a58917c05b54337b19e8ccc493f4d1\": container with ID starting with dc3827cbe60126b1861248a10bd8def727a58917c05b54337b19e8ccc493f4d1 not found: ID does not exist" containerID="dc3827cbe60126b1861248a10bd8def727a58917c05b54337b19e8ccc493f4d1"
Sep 04 23:53:17.614475 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.614387    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc3827cbe60126b1861248a10bd8def727a58917c05b54337b19e8ccc493f4d1"} err="failed to get container status \"dc3827cbe60126b1861248a10bd8def727a58917c05b54337b19e8ccc493f4d1\": rpc error: code = NotFound desc = could not find container \"dc3827cbe60126b1861248a10bd8def727a58917c05b54337b19e8ccc493f4d1\": container with ID starting with dc3827cbe60126b1861248a10bd8def727a58917c05b54337b19e8ccc493f4d1 not found: ID does not exist"
Sep 04 23:53:17.644028 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.643987    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"]
Sep 04 23:53:17.650538 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:17.650510    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f4c99d38-788689fbbf-2xn44"]
Sep 04 23:53:18.358342 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:18.358299    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="180491bd-e5f4-4ed6-850b-a20d8dde5695" path="/var/lib/kubelet/pods/180491bd-e5f4-4ed6-850b-a20d8dde5695/volumes"
Sep 04 23:53:19.862521 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:19.862485    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"]
Sep 04 23:53:19.863699 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:19.863672    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="180491bd-e5f4-4ed6-850b-a20d8dde5695" containerName="maas-api"
Sep 04 23:53:19.863699 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:19.863699    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="180491bd-e5f4-4ed6-850b-a20d8dde5695" containerName="maas-api"
Sep 04 23:53:19.863880 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:19.863798    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="180491bd-e5f4-4ed6-850b-a20d8dde5695" containerName="maas-api"
Sep 04 23:53:19.868763 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:19.868739    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:19.871143 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:19.871114    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mnd5p\""
Sep 04 23:53:19.871143 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:19.871134    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-5245c1-serving-cert\""
Sep 04 23:53:19.875957 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:19.875578    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"]
Sep 04 23:53:19.999198 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:19.999158    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f383e85-8df3-4386-90f2-75d57fa6b922-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:19.999735 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:19.999710    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f383e85-8df3-4386-90f2-75d57fa6b922-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:19.999998 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:19.999978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f383e85-8df3-4386-90f2-75d57fa6b922-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:20.000258 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:20.000233    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w886\" (UniqueName: \"kubernetes.io/projected/6f383e85-8df3-4386-90f2-75d57fa6b922-kube-api-access-8w886\") pod \"maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:20.100762 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:20.100725    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w886\" (UniqueName: \"kubernetes.io/projected/6f383e85-8df3-4386-90f2-75d57fa6b922-kube-api-access-8w886\") pod \"maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:20.100961 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:20.100788    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f383e85-8df3-4386-90f2-75d57fa6b922-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:20.100961 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:20.100808    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f383e85-8df3-4386-90f2-75d57fa6b922-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:20.100961 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:20.100845    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f383e85-8df3-4386-90f2-75d57fa6b922-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:20.101592 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:20.101556    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f383e85-8df3-4386-90f2-75d57fa6b922-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:20.103260 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:20.103240    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f383e85-8df3-4386-90f2-75d57fa6b922-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:20.103361 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:20.103296    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f383e85-8df3-4386-90f2-75d57fa6b922-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:20.107822 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:20.107800    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w886\" (UniqueName: \"kubernetes.io/projected/6f383e85-8df3-4386-90f2-75d57fa6b922-kube-api-access-8w886\") pod \"maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:20.184386 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:20.184308    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:20.472559 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:20.472534    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"]
Sep 04 23:53:20.474450 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:53:20.474420    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f383e85_8df3_4386_90f2_75d57fa6b922.slice/crio-101bc2e618a9827a51aa1653000ceb7ed59272f0578d1dfb7458b7d63fee2254 WatchSource:0}: Error finding container 101bc2e618a9827a51aa1653000ceb7ed59272f0578d1dfb7458b7d63fee2254: Status 404 returned error can't find the container with id 101bc2e618a9827a51aa1653000ceb7ed59272f0578d1dfb7458b7d63fee2254
Sep 04 23:53:20.622371 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:20.622331    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr" event={"ID":"6f383e85-8df3-4386-90f2-75d57fa6b922","Type":"ContainerStarted","Data":"101bc2e618a9827a51aa1653000ceb7ed59272f0578d1dfb7458b7d63fee2254"}
Sep 04 23:53:21.629664 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:21.629562    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr" event={"ID":"6f383e85-8df3-4386-90f2-75d57fa6b922","Type":"ContainerStarted","Data":"4e5da510b4a11e51f71e9178da18cb7c036038414f5984531335b19a3e2bd4e7"}
Sep 04 23:53:21.630029 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:21.629723    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:21.645892 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:21.645848    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr" podStartSLOduration=1.80124891 podStartE2EDuration="2.64583411s" totalImagesPullingTime="844.5852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:53:21.644623655 +0000 UTC m=+1013.891290733" watchObservedRunningTime="2026-09-04 23:53:21.64583411 +0000 UTC m=+1013.892501188"
Sep 04 23:53:27.642020 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:27.641993    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 04 23:53:33.646834 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:33.646788    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 04 23:53:33.647205 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:33.646861    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf" podUID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 04 23:53:34.700995 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:34.700899    2562 generic.go:356] "Generic (PLEG): container finished" podID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" containerID="26ba251087b4b397ad44c4de837187ac8b0fc24e52542773c289e0be4c869045" exitCode=0
Sep 04 23:53:34.700995 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:34.700936    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf" event={"ID":"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de","Type":"ContainerDied","Data":"26ba251087b4b397ad44c4de837187ac8b0fc24e52542773c289e0be4c869045"}
Sep 04 23:53:34.701419 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:34.701309    2562 scope.go:117] "RemoveContainer" containerID="26ba251087b4b397ad44c4de837187ac8b0fc24e52542773c289e0be4c869045"
Sep 04 23:53:34.702464 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:34.702431    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:53:34.702528 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:34.702517    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:53:34.702597 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:34.702583    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:53:35.708878 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:35.708844    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf" event={"ID":"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de","Type":"ContainerStarted","Data":"45ff9fc9cb186140398b1e40961ea2638d32812e91603f2e1e484b8dc60e23c6"}
Sep 04 23:53:36.050788 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:36.050694    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:36.050976 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:36.050917    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 04 23:53:36.051049 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:36.050973    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf" podUID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 04 23:53:37.050559 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:37.050517    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 04 23:53:37.051000 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:37.050583    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf" podUID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 04 23:53:38.050813 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:38.050774    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 04 23:53:38.051213 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:38.050840    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf" podUID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 04 23:53:39.050347 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:39.050301    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 04 23:53:39.050557 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:39.050375    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf" podUID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 04 23:53:40.050507 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.050463    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 04 23:53:40.051039 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.050523    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf" podUID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 04 23:53:40.414323 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.414238    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"]
Sep 04 23:53:40.428686 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.426612    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:40.436807 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.433771    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-e35897-serving-cert\""
Sep 04 23:53:40.436807 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.435074    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"]
Sep 04 23:53:40.491028 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.490988    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40083a10-52ff-4d73-8f77-32ca1ea66111-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:40.491213 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.491152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40083a10-52ff-4d73-8f77-32ca1ea66111-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:40.491213 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.491186    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40083a10-52ff-4d73-8f77-32ca1ea66111-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:40.491331 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.491219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q5wp\" (UniqueName: \"kubernetes.io/projected/40083a10-52ff-4d73-8f77-32ca1ea66111-kube-api-access-8q5wp\") pod \"maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:40.591989 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.591948    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40083a10-52ff-4d73-8f77-32ca1ea66111-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:40.592167 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.592053    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40083a10-52ff-4d73-8f77-32ca1ea66111-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:40.592167 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.592078    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40083a10-52ff-4d73-8f77-32ca1ea66111-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:40.592167 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.592101    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q5wp\" (UniqueName: \"kubernetes.io/projected/40083a10-52ff-4d73-8f77-32ca1ea66111-kube-api-access-8q5wp\") pod \"maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:40.592830 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.592803    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40083a10-52ff-4d73-8f77-32ca1ea66111-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:40.594519 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.594485    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40083a10-52ff-4d73-8f77-32ca1ea66111-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:40.594636 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.594615    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40083a10-52ff-4d73-8f77-32ca1ea66111-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:40.599481 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.599463    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q5wp\" (UniqueName: \"kubernetes.io/projected/40083a10-52ff-4d73-8f77-32ca1ea66111-kube-api-access-8q5wp\") pod \"maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:40.761804 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:40.761764    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:41.053770 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:41.053688    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:41.054116 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:41.053959    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:41.054609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:41.054588    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 04 23:53:41.057908 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:41.057870    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"]
Sep 04 23:53:41.061055 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:53:41.061026    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40083a10_52ff_4d73_8f77_32ca1ea66111.slice/crio-7def5ae806213bb773ee6f7b87c4d1ea9000cdb1ca9746cc78e03379b0ca6733 WatchSource:0}: Error finding container 7def5ae806213bb773ee6f7b87c4d1ea9000cdb1ca9746cc78e03379b0ca6733: Status 404 returned error can't find the container with id 7def5ae806213bb773ee6f7b87c4d1ea9000cdb1ca9746cc78e03379b0ca6733
Sep 04 23:53:41.760748 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:41.760691    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8" event={"ID":"40083a10-52ff-4d73-8f77-32ca1ea66111","Type":"ContainerStarted","Data":"41dac8f3897fd3315dbd8432f18aeda8bfaec029d036ba6d1fcee7cc82799556"}
Sep 04 23:53:41.760947 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:41.760775    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8" event={"ID":"40083a10-52ff-4d73-8f77-32ca1ea66111","Type":"ContainerStarted","Data":"7def5ae806213bb773ee6f7b87c4d1ea9000cdb1ca9746cc78e03379b0ca6733"}
Sep 04 23:53:41.761091 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:41.761065    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:41.779556 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:41.779509    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8" podStartSLOduration=1.180581382 podStartE2EDuration="1.779492347s" totalImagesPullingTime="598.910965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:53:41.778445499 +0000 UTC m=+1034.025112578" watchObservedRunningTime="2026-09-04 23:53:41.779492347 +0000 UTC m=+1034.026159427"
Sep 04 23:53:47.771743 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:47.771714    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:53:48.411828 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:48.411797    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"]
Sep 04 23:53:48.414610 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:48.414595    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:48.417004 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:48.416981    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-88828d-serving-cert\""
Sep 04 23:53:48.427297 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:48.427272    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"]
Sep 04 23:53:48.466748 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:48.466716    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:48.466952 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:48.466755    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:48.466952 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:48.466799    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb8h2\" (UniqueName: \"kubernetes.io/projected/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-kube-api-access-hb8h2\") pod \"maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:48.466952 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:48.466875    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:48.567555 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:48.567517    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:48.567786 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:48.567616    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:48.567786 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:48.567639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:48.567786 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:48.567678    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8h2\" (UniqueName: \"kubernetes.io/projected/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-kube-api-access-hb8h2\") pod \"maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:48.567786 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:53:48.567718    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-serving-cert: secret "maas-api-e2e-shared-b-w3-88828d-serving-cert" not found
Sep 04 23:53:48.568013 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:53:48.567798    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-maas-api-tls podName:c1faaa8f-26fc-4058-b7ee-7a5e7356075a nodeName:}" failed. No retries permitted until 2026-09-04 23:53:49.067779257 +0000 UTC m=+1041.314446332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-maas-api-tls") pod "maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp" (UID: "c1faaa8f-26fc-4058-b7ee-7a5e7356075a") : secret "maas-api-e2e-shared-b-w3-88828d-serving-cert" not found
Sep 04 23:53:48.568329 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:48.568309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:48.570090 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:48.570061    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:48.578714 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:48.578689    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb8h2\" (UniqueName: \"kubernetes.io/projected/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-kube-api-access-hb8h2\") pod \"maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:49.073481 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:49.073438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:49.075697 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:49.075670    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:49.327842 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:49.327754    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:49.599274 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:49.599249    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"]
Sep 04 23:53:49.600837 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:53:49.600810    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1faaa8f_26fc_4058_b7ee_7a5e7356075a.slice/crio-9c205f4860c621eca46750c18b7fb929f9537c2bb272eaff119029595387aaac WatchSource:0}: Error finding container 9c205f4860c621eca46750c18b7fb929f9537c2bb272eaff119029595387aaac: Status 404 returned error can't find the container with id 9c205f4860c621eca46750c18b7fb929f9537c2bb272eaff119029595387aaac
Sep 04 23:53:49.805773 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:49.805738    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp" event={"ID":"c1faaa8f-26fc-4058-b7ee-7a5e7356075a","Type":"ContainerStarted","Data":"9c205f4860c621eca46750c18b7fb929f9537c2bb272eaff119029595387aaac"}
Sep 04 23:53:50.812349 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:50.812314    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp" event={"ID":"c1faaa8f-26fc-4058-b7ee-7a5e7356075a","Type":"ContainerStarted","Data":"ff55d3c830b8e0d9636e3958d78f2487854826e4c589a0b3f5241aa35e375bc5"}
Sep 04 23:53:50.812765 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:50.812480    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:53:50.831401 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:50.831352    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp" podStartSLOduration=2.52348456 podStartE2EDuration="2.83133613s" totalImagesPullingTime="307.85157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:53:50.82878864 +0000 UTC m=+1043.075455728" watchObservedRunningTime="2026-09-04 23:53:50.83133613 +0000 UTC m=+1043.078003210"
Sep 04 23:53:56.822009 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:53:56.821978    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 04 23:54:08.684521 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:08.684479    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"]
Sep 04 23:54:08.688454 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:08.688430    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:08.691447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:08.691416    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-f7eb8e-serving-cert\""
Sep 04 23:54:08.700455 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:08.697238    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"]
Sep 04 23:54:08.873484 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:08.873447    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b62954-f509-4383-8796-21883a065c05-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:08.873637 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:08.873554    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b62954-f509-4383-8796-21883a065c05-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:08.873637 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:08.873594    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b62954-f509-4383-8796-21883a065c05-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:08.873778 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:08.873723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmwj8\" (UniqueName: \"kubernetes.io/projected/66b62954-f509-4383-8796-21883a065c05-kube-api-access-bmwj8\") pod \"maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:08.974450 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:08.974412    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b62954-f509-4383-8796-21883a065c05-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:08.974718 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:08.974521    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmwj8\" (UniqueName: \"kubernetes.io/projected/66b62954-f509-4383-8796-21883a065c05-kube-api-access-bmwj8\") pod \"maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:08.974718 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:08.974571    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b62954-f509-4383-8796-21883a065c05-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:08.974718 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:54:08.974579    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-serving-cert: secret "maas-api-e2e-shared-b-w2-f7eb8e-serving-cert" not found
Sep 04 23:54:08.974718 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:08.974639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b62954-f509-4383-8796-21883a065c05-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:08.974718 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:54:08.974685    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b62954-f509-4383-8796-21883a065c05-maas-api-tls podName:66b62954-f509-4383-8796-21883a065c05 nodeName:}" failed. No retries permitted until 2026-09-04 23:54:09.474636256 +0000 UTC m=+1061.721303331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66b62954-f509-4383-8796-21883a065c05-maas-api-tls") pod "maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw" (UID: "66b62954-f509-4383-8796-21883a065c05") : secret "maas-api-e2e-shared-b-w2-f7eb8e-serving-cert" not found
Sep 04 23:54:08.975318 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:08.975296    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b62954-f509-4383-8796-21883a065c05-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:08.977171 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:08.977148    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b62954-f509-4383-8796-21883a065c05-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:08.982343 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:08.982322    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmwj8\" (UniqueName: \"kubernetes.io/projected/66b62954-f509-4383-8796-21883a065c05-kube-api-access-bmwj8\") pod \"maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:09.479667 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:09.479608    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b62954-f509-4383-8796-21883a065c05-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:09.482038 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:09.482017    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b62954-f509-4383-8796-21883a065c05-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:09.603664 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:09.603629    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:09.879894 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:09.879863    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"]
Sep 04 23:54:09.881752 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:54:09.881725    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b62954_f509_4383_8796_21883a065c05.slice/crio-1fe1f726c08abf7120d590ce276979df4a2a61e130d61e08c5b6793bfe01e9bc WatchSource:0}: Error finding container 1fe1f726c08abf7120d590ce276979df4a2a61e130d61e08c5b6793bfe01e9bc: Status 404 returned error can't find the container with id 1fe1f726c08abf7120d590ce276979df4a2a61e130d61e08c5b6793bfe01e9bc
Sep 04 23:54:09.925584 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:09.925551    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw" event={"ID":"66b62954-f509-4383-8796-21883a065c05","Type":"ContainerStarted","Data":"1fe1f726c08abf7120d590ce276979df4a2a61e130d61e08c5b6793bfe01e9bc"}
Sep 04 23:54:11.938225 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:11.938182    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw" event={"ID":"66b62954-f509-4383-8796-21883a065c05","Type":"ContainerStarted","Data":"ad60af856720cd8b1d1306173e5734d451039efa8dc1cea6e5f72714f0538b0d"}
Sep 04 23:54:11.938665 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:11.938337    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:11.954347 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:11.954286    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw" podStartSLOduration=2.666645549 podStartE2EDuration="3.954270403s" totalImagesPullingTime="1.287624854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:54:11.952959811 +0000 UTC m=+1064.199626889" watchObservedRunningTime="2026-09-04 23:54:11.954270403 +0000 UTC m=+1064.200937504"
Sep 04 23:54:17.949322 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:17.949290    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 04 23:54:45.788688 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:45.788475    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"]
Sep 04 23:54:45.792551 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:45.792527    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:45.795115 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:45.795089    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ede0daf1-serving-cert\""
Sep 04 23:54:45.805689 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:45.805664    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"]
Sep 04 23:54:45.934159 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:45.934124    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07038953-c676-46f9-aa63-fd6be10d5850-maas-api-tls\") pod \"maas-api-e2e-mt-ede0daf1-667dc766c-vrldt\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:45.934325 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:45.934235    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07038953-c676-46f9-aa63-fd6be10d5850-openshift-service-ca\") pod \"maas-api-e2e-mt-ede0daf1-667dc766c-vrldt\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:45.934325 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:45.934261    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4csnh\" (UniqueName: \"kubernetes.io/projected/07038953-c676-46f9-aa63-fd6be10d5850-kube-api-access-4csnh\") pod \"maas-api-e2e-mt-ede0daf1-667dc766c-vrldt\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:45.934325 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:45.934286    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07038953-c676-46f9-aa63-fd6be10d5850-metrics-certs\") pod \"maas-api-e2e-mt-ede0daf1-667dc766c-vrldt\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:46.035567 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:46.035522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07038953-c676-46f9-aa63-fd6be10d5850-maas-api-tls\") pod \"maas-api-e2e-mt-ede0daf1-667dc766c-vrldt\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:46.035783 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:46.035606    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07038953-c676-46f9-aa63-fd6be10d5850-openshift-service-ca\") pod \"maas-api-e2e-mt-ede0daf1-667dc766c-vrldt\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:46.035783 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:46.035629    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4csnh\" (UniqueName: \"kubernetes.io/projected/07038953-c676-46f9-aa63-fd6be10d5850-kube-api-access-4csnh\") pod \"maas-api-e2e-mt-ede0daf1-667dc766c-vrldt\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:46.035783 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:46.035645    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07038953-c676-46f9-aa63-fd6be10d5850-metrics-certs\") pod \"maas-api-e2e-mt-ede0daf1-667dc766c-vrldt\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:46.035783 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:54:46.035724    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-serving-cert: secret "maas-api-e2e-mt-ede0daf1-serving-cert" not found
Sep 04 23:54:46.036008 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:54:46.035803    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07038953-c676-46f9-aa63-fd6be10d5850-maas-api-tls podName:07038953-c676-46f9-aa63-fd6be10d5850 nodeName:}" failed. No retries permitted until 2026-09-04 23:54:46.535782775 +0000 UTC m=+1098.782449832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07038953-c676-46f9-aa63-fd6be10d5850-maas-api-tls") pod "maas-api-e2e-mt-ede0daf1-667dc766c-vrldt" (UID: "07038953-c676-46f9-aa63-fd6be10d5850") : secret "maas-api-e2e-mt-ede0daf1-serving-cert" not found
Sep 04 23:54:46.036420 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:46.036393    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07038953-c676-46f9-aa63-fd6be10d5850-openshift-service-ca\") pod \"maas-api-e2e-mt-ede0daf1-667dc766c-vrldt\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:46.038213 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:46.038193    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07038953-c676-46f9-aa63-fd6be10d5850-metrics-certs\") pod \"maas-api-e2e-mt-ede0daf1-667dc766c-vrldt\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:46.044635 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:46.044582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4csnh\" (UniqueName: \"kubernetes.io/projected/07038953-c676-46f9-aa63-fd6be10d5850-kube-api-access-4csnh\") pod \"maas-api-e2e-mt-ede0daf1-667dc766c-vrldt\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:46.540846 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:46.540814    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07038953-c676-46f9-aa63-fd6be10d5850-maas-api-tls\") pod \"maas-api-e2e-mt-ede0daf1-667dc766c-vrldt\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:46.543317 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:46.543292    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07038953-c676-46f9-aa63-fd6be10d5850-maas-api-tls\") pod \"maas-api-e2e-mt-ede0daf1-667dc766c-vrldt\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:46.707404 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:46.707369    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:46.980898 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:46.980873    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"]
Sep 04 23:54:46.982709 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:54:46.982683    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07038953_c676_46f9_aa63_fd6be10d5850.slice/crio-3943baa39119d89a8b300633b35bbecda71375a59e090cb4065132a4b12b5709 WatchSource:0}: Error finding container 3943baa39119d89a8b300633b35bbecda71375a59e090cb4065132a4b12b5709: Status 404 returned error can't find the container with id 3943baa39119d89a8b300633b35bbecda71375a59e090cb4065132a4b12b5709
Sep 04 23:54:47.132420 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:47.132388    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt" event={"ID":"07038953-c676-46f9-aa63-fd6be10d5850","Type":"ContainerStarted","Data":"3943baa39119d89a8b300633b35bbecda71375a59e090cb4065132a4b12b5709"}
Sep 04 23:54:48.139320 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:48.139288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt" event={"ID":"07038953-c676-46f9-aa63-fd6be10d5850","Type":"ContainerStarted","Data":"ab08d0e0c9a0c0135e825631c556b83687aa509891617ae1dac14de632b19283"}
Sep 04 23:54:48.139715 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:48.139351    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:54:48.154965 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:48.154922    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt" podStartSLOduration=2.453334832 podStartE2EDuration="3.154906887s" totalImagesPullingTime="701.572055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:54:48.153756935 +0000 UTC m=+1100.400424019" watchObservedRunningTime="2026-09-04 23:54:48.154906887 +0000 UTC m=+1100.401573966"
Sep 04 23:54:54.151311 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:54:54.151276    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:56:14.791155 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:14.791108    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"]
Sep 04 23:56:14.791761 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:14.791358    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt" podUID="07038953-c676-46f9-aa63-fd6be10d5850" containerName="maas-api" containerID="cri-o://ab08d0e0c9a0c0135e825631c556b83687aa509891617ae1dac14de632b19283" gracePeriod=30
Sep 04 23:56:15.049702 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.049619    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:56:15.196729 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.196696    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07038953-c676-46f9-aa63-fd6be10d5850-openshift-service-ca\") pod \"07038953-c676-46f9-aa63-fd6be10d5850\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") "
Sep 04 23:56:15.196896 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.196755    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07038953-c676-46f9-aa63-fd6be10d5850-maas-api-tls\") pod \"07038953-c676-46f9-aa63-fd6be10d5850\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") "
Sep 04 23:56:15.196896 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.196814    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07038953-c676-46f9-aa63-fd6be10d5850-metrics-certs\") pod \"07038953-c676-46f9-aa63-fd6be10d5850\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") "
Sep 04 23:56:15.196896 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.196852    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4csnh\" (UniqueName: \"kubernetes.io/projected/07038953-c676-46f9-aa63-fd6be10d5850-kube-api-access-4csnh\") pod \"07038953-c676-46f9-aa63-fd6be10d5850\" (UID: \"07038953-c676-46f9-aa63-fd6be10d5850\") "
Sep 04 23:56:15.197152 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.197070    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07038953-c676-46f9-aa63-fd6be10d5850-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07038953-c676-46f9-aa63-fd6be10d5850" (UID: "07038953-c676-46f9-aa63-fd6be10d5850"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:56:15.198966 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.198931    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07038953-c676-46f9-aa63-fd6be10d5850-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "07038953-c676-46f9-aa63-fd6be10d5850" (UID: "07038953-c676-46f9-aa63-fd6be10d5850"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:56:15.199072 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.198974    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07038953-c676-46f9-aa63-fd6be10d5850-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07038953-c676-46f9-aa63-fd6be10d5850" (UID: "07038953-c676-46f9-aa63-fd6be10d5850"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:56:15.199072 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.199020    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07038953-c676-46f9-aa63-fd6be10d5850-kube-api-access-4csnh" (OuterVolumeSpecName: "kube-api-access-4csnh") pod "07038953-c676-46f9-aa63-fd6be10d5850" (UID: "07038953-c676-46f9-aa63-fd6be10d5850"). InnerVolumeSpecName "kube-api-access-4csnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:56:15.298047 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.297998    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07038953-c676-46f9-aa63-fd6be10d5850-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:56:15.298047 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.298043    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07038953-c676-46f9-aa63-fd6be10d5850-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:56:15.298047 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.298054    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4csnh\" (UniqueName: \"kubernetes.io/projected/07038953-c676-46f9-aa63-fd6be10d5850-kube-api-access-4csnh\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:56:15.298047 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.298064    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07038953-c676-46f9-aa63-fd6be10d5850-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:56:15.613906 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.613858    2562 generic.go:356] "Generic (PLEG): container finished" podID="07038953-c676-46f9-aa63-fd6be10d5850" containerID="ab08d0e0c9a0c0135e825631c556b83687aa509891617ae1dac14de632b19283" exitCode=0
Sep 04 23:56:15.614109 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.613997    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt" event={"ID":"07038953-c676-46f9-aa63-fd6be10d5850","Type":"ContainerDied","Data":"ab08d0e0c9a0c0135e825631c556b83687aa509891617ae1dac14de632b19283"}
Sep 04 23:56:15.614109 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.614086    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt" event={"ID":"07038953-c676-46f9-aa63-fd6be10d5850","Type":"ContainerDied","Data":"3943baa39119d89a8b300633b35bbecda71375a59e090cb4065132a4b12b5709"}
Sep 04 23:56:15.614235 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.614126    2562 scope.go:117] "RemoveContainer" containerID="ab08d0e0c9a0c0135e825631c556b83687aa509891617ae1dac14de632b19283"
Sep 04 23:56:15.614387 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.614370    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"
Sep 04 23:56:15.624019 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.624001    2562 scope.go:117] "RemoveContainer" containerID="ab08d0e0c9a0c0135e825631c556b83687aa509891617ae1dac14de632b19283"
Sep 04 23:56:15.624302 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:56:15.624283    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab08d0e0c9a0c0135e825631c556b83687aa509891617ae1dac14de632b19283\": container with ID starting with ab08d0e0c9a0c0135e825631c556b83687aa509891617ae1dac14de632b19283 not found: ID does not exist" containerID="ab08d0e0c9a0c0135e825631c556b83687aa509891617ae1dac14de632b19283"
Sep 04 23:56:15.624357 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.624311    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab08d0e0c9a0c0135e825631c556b83687aa509891617ae1dac14de632b19283"} err="failed to get container status \"ab08d0e0c9a0c0135e825631c556b83687aa509891617ae1dac14de632b19283\": rpc error: code = NotFound desc = could not find container \"ab08d0e0c9a0c0135e825631c556b83687aa509891617ae1dac14de632b19283\": container with ID starting with ab08d0e0c9a0c0135e825631c556b83687aa509891617ae1dac14de632b19283 not found: ID does not exist"
Sep 04 23:56:15.636448 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.636420    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"]
Sep 04 23:56:15.638379 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:15.638360    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ede0daf1-667dc766c-vrldt"]
Sep 04 23:56:16.358858 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:16.358827    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07038953-c676-46f9-aa63-fd6be10d5850" path="/var/lib/kubelet/pods/07038953-c676-46f9-aa63-fd6be10d5850/volumes"
Sep 04 23:56:28.271020 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:28.270991    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 04 23:56:28.274960 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:28.274938    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 04 23:56:28.275157 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:28.275138    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 04 23:56:28.279209 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:28.279192    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 04 23:56:28.375037 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:56:28.375010    2562 scope.go:117] "RemoveContainer" containerID="b440df059e3c85f209266d83715c54324dc80884f0a896ba07ef126887cb5ab6"
Sep 04 23:58:04.038681 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.038619    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"]
Sep 04 23:58:04.039323 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.039304    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07038953-c676-46f9-aa63-fd6be10d5850" containerName="maas-api"
Sep 04 23:58:04.039370 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.039326    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="07038953-c676-46f9-aa63-fd6be10d5850" containerName="maas-api"
Sep 04 23:58:04.039444 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.039432    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="07038953-c676-46f9-aa63-fd6be10d5850" containerName="maas-api"
Sep 04 23:58:04.043300 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.043263    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:04.045868 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.045643    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-87b0d1-serving-cert\""
Sep 04 23:58:04.049222 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.049197    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"]
Sep 04 23:58:04.142714 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.142673    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6200ed10-cb40-43c9-b9cd-679057437ade-maas-api-tls\") pod \"maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:04.142928 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.142753    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6200ed10-cb40-43c9-b9cd-679057437ade-metrics-certs\") pod \"maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:04.142928 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.142802    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqhnz\" (UniqueName: \"kubernetes.io/projected/6200ed10-cb40-43c9-b9cd-679057437ade-kube-api-access-mqhnz\") pod \"maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:04.142928 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.142849    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6200ed10-cb40-43c9-b9cd-679057437ade-openshift-service-ca\") pod \"maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:04.243419 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.243378    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqhnz\" (UniqueName: \"kubernetes.io/projected/6200ed10-cb40-43c9-b9cd-679057437ade-kube-api-access-mqhnz\") pod \"maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:04.243609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.243430    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6200ed10-cb40-43c9-b9cd-679057437ade-openshift-service-ca\") pod \"maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:04.243609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.243519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6200ed10-cb40-43c9-b9cd-679057437ade-maas-api-tls\") pod \"maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:04.243609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.243583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6200ed10-cb40-43c9-b9cd-679057437ade-metrics-certs\") pod \"maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:04.243795 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:58:04.243628    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-serving-cert: secret "maas-api-e2e-api-a-87b0d1-serving-cert" not found
Sep 04 23:58:04.243795 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:58:04.243722    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6200ed10-cb40-43c9-b9cd-679057437ade-maas-api-tls podName:6200ed10-cb40-43c9-b9cd-679057437ade nodeName:}" failed. No retries permitted until 2026-09-04 23:58:04.743700282 +0000 UTC m=+1296.990367356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6200ed10-cb40-43c9-b9cd-679057437ade-maas-api-tls") pod "maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm" (UID: "6200ed10-cb40-43c9-b9cd-679057437ade") : secret "maas-api-e2e-api-a-87b0d1-serving-cert" not found
Sep 04 23:58:04.244231 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.244204    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6200ed10-cb40-43c9-b9cd-679057437ade-openshift-service-ca\") pod \"maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:04.246154 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.246122    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6200ed10-cb40-43c9-b9cd-679057437ade-metrics-certs\") pod \"maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:04.252141 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.252111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqhnz\" (UniqueName: \"kubernetes.io/projected/6200ed10-cb40-43c9-b9cd-679057437ade-kube-api-access-mqhnz\") pod \"maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:04.748537 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.748499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6200ed10-cb40-43c9-b9cd-679057437ade-maas-api-tls\") pod \"maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:04.751627 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.751587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6200ed10-cb40-43c9-b9cd-679057437ade-maas-api-tls\") pod \"maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:04.956892 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:04.956862    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:05.230542 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:05.230516    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"]
Sep 04 23:58:05.232311 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:58:05.232287    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6200ed10_cb40_43c9_b9cd_679057437ade.slice/crio-0ad8b90f37fce47fa777db68472ac0f73cdcc4488d8a98156a3562926877f3d3 WatchSource:0}: Error finding container 0ad8b90f37fce47fa777db68472ac0f73cdcc4488d8a98156a3562926877f3d3: Status 404 returned error can't find the container with id 0ad8b90f37fce47fa777db68472ac0f73cdcc4488d8a98156a3562926877f3d3
Sep 04 23:58:05.233609 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:05.233592    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 23:58:06.207761 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:06.207715    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm" event={"ID":"6200ed10-cb40-43c9-b9cd-679057437ade","Type":"ContainerStarted","Data":"3a1e2e72b228e2e172bcabb0c422b05d7d1dddf9f22db83234ce62572d0f6ca9"}
Sep 04 23:58:06.207961 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:06.207888    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:06.207961 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:06.207912    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm" event={"ID":"6200ed10-cb40-43c9-b9cd-679057437ade","Type":"ContainerStarted","Data":"0ad8b90f37fce47fa777db68472ac0f73cdcc4488d8a98156a3562926877f3d3"}
Sep 04 23:58:06.224587 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:06.224532    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm" podStartSLOduration=1.7561198070000001 podStartE2EDuration="2.224516625s" totalImagesPullingTime="468.396818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:58:06.222264815 +0000 UTC m=+1298.468931894" watchObservedRunningTime="2026-09-04 23:58:06.224516625 +0000 UTC m=+1298.471183705"
Sep 04 23:58:12.219196 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:12.219165    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:58:28.555453 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.555406    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"]
Sep 04 23:58:28.559317 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.559291    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.561643 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.561619    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8cff3b-openshift-default-dockercfg-p8skf\""
Sep 04 23:58:28.571544 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.571516    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"]
Sep 04 23:58:28.586591 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.586561    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7kjs\" (UniqueName: \"kubernetes.io/projected/59f9eb01-4423-4f9a-9e53-86fc485ddeec-kube-api-access-v7kjs\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.586779 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.586608    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-workload-certs\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.586779 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.586711    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-token\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.586779 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.586752    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-envoy\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.586969 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.586798    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-workload-socket\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.586969 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.586834    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-credential-socket\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.586969 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.586872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-podinfo\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.586969 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.586951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istiod-ca-cert\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.587133 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.586991    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-data\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.688265 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.688229    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-podinfo\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.688463 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.688288    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istiod-ca-cert\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.688463 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.688318    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-data\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.688463 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.688346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7kjs\" (UniqueName: \"kubernetes.io/projected/59f9eb01-4423-4f9a-9e53-86fc485ddeec-kube-api-access-v7kjs\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.688463 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.688371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-workload-certs\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.688463 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.688420    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-token\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.688463 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.688451    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-envoy\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.688837 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.688488    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-workload-socket\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.688837 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.688710    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-credential-socket\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.688928 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.688842    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-workload-certs\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.688928 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.688904    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-workload-socket\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.689038 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.688945    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-data\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.689094 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.689039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-credential-socket\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.689347 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.689322    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istiod-ca-cert\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.690718 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.690701    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-envoy\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.690956 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.690938    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-podinfo\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.702116 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.702080    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-token\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.702425 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.702402    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7kjs\" (UniqueName: \"kubernetes.io/projected/59f9eb01-4423-4f9a-9e53-86fc485ddeec-kube-api-access-v7kjs\") pod \"e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") " pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:28.871265 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:28.871170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:29.146780 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:29.146750    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"]
Sep 04 23:58:29.148671 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:58:29.148621    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f9eb01_4423_4f9a_9e53_86fc485ddeec.slice/crio-675875967f3eb2d9a55b6d65d6849ed9ca683aa122e21538be8ada5c0cee1084 WatchSource:0}: Error finding container 675875967f3eb2d9a55b6d65d6849ed9ca683aa122e21538be8ada5c0cee1084: Status 404 returned error can't find the container with id 675875967f3eb2d9a55b6d65d6849ed9ca683aa122e21538be8ada5c0cee1084
Sep 04 23:58:29.151066 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:29.151033    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:58:29.151138 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:29.151100    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:58:29.151138 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:29.151131    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:58:29.335089 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:29.335051    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp" event={"ID":"59f9eb01-4423-4f9a-9e53-86fc485ddeec","Type":"ContainerStarted","Data":"d8cfea487c9e0309649d1304f1534d355cda89ac16c7ae89ee1ced33d41554e2"}
Sep 04 23:58:29.335261 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:29.335119    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp" event={"ID":"59f9eb01-4423-4f9a-9e53-86fc485ddeec","Type":"ContainerStarted","Data":"675875967f3eb2d9a55b6d65d6849ed9ca683aa122e21538be8ada5c0cee1084"}
Sep 04 23:58:29.353885 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:29.353827    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp" podStartSLOduration=1.353811214 podStartE2EDuration="1.353811214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:58:29.35136018 +0000 UTC m=+1321.598027259" watchObservedRunningTime="2026-09-04 23:58:29.353811214 +0000 UTC m=+1321.600478292"
Sep 04 23:58:29.871393 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:29.871358    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:30.877547 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:30.877512    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:31.347978 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:31.347948    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:31.349041 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:31.349020    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:32.315558 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:32.315527    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"]
Sep 04 23:58:32.320597 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:32.320572    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:32.322827 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:32.322802    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8cff3b-serving-cert\""
Sep 04 23:58:32.330241 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:32.330216    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"]
Sep 04 23:58:32.425982 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:32.425942    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-maas-api-tls\") pod \"maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:32.426177 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:32.426055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699fa135-23af-478d-a75d-11518425efd2-openshift-service-ca\") pod \"maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:32.426177 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:32.426107    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cstnr\" (UniqueName: \"kubernetes.io/projected/699fa135-23af-478d-a75d-11518425efd2-kube-api-access-cstnr\") pod \"maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:32.426177 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:32.426159    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-metrics-certs\") pod \"maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:32.526823 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:32.526781    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cstnr\" (UniqueName: \"kubernetes.io/projected/699fa135-23af-478d-a75d-11518425efd2-kube-api-access-cstnr\") pod \"maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:32.526988 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:32.526949    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-metrics-certs\") pod \"maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:32.527094 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:32.527076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-maas-api-tls\") pod \"maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:32.527204 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:58:32.527187    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-serving-cert: secret "maas-api-e2e-api-b-8cff3b-serving-cert" not found
Sep 04 23:58:32.527255 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:58:32.527245    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-maas-api-tls podName:699fa135-23af-478d-a75d-11518425efd2 nodeName:}" failed. No retries permitted until 2026-09-04 23:58:33.027229487 +0000 UTC m=+1325.273896547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-maas-api-tls") pod "maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf" (UID: "699fa135-23af-478d-a75d-11518425efd2") : secret "maas-api-e2e-api-b-8cff3b-serving-cert" not found
Sep 04 23:58:32.527314 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:32.527191    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699fa135-23af-478d-a75d-11518425efd2-openshift-service-ca\") pod \"maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:32.527886 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:32.527868    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699fa135-23af-478d-a75d-11518425efd2-openshift-service-ca\") pod \"maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:32.529335 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:32.529313    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-metrics-certs\") pod \"maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:32.544872 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:32.544850    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cstnr\" (UniqueName: \"kubernetes.io/projected/699fa135-23af-478d-a75d-11518425efd2-kube-api-access-cstnr\") pod \"maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:33.033443 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:33.033412    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-maas-api-tls\") pod \"maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:33.033622 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:58:33.033558    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-serving-cert: secret "maas-api-e2e-api-b-8cff3b-serving-cert" not found
Sep 04 23:58:33.033622 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:58:33.033618    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-maas-api-tls podName:699fa135-23af-478d-a75d-11518425efd2 nodeName:}" failed. No retries permitted until 2026-09-04 23:58:34.033601872 +0000 UTC m=+1326.280268942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-maas-api-tls") pod "maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf" (UID: "699fa135-23af-478d-a75d-11518425efd2") : secret "maas-api-e2e-api-b-8cff3b-serving-cert" not found
Sep 04 23:58:34.045131 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:34.045088    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-maas-api-tls\") pod \"maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:34.047706 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:34.047671    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-maas-api-tls\") pod \"maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:34.137150 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:34.137108    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:34.432063 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:34.414073    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"]
Sep 04 23:58:34.432063 ip-10-0-138-66 kubenswrapper[2562]: W0904 23:58:34.415250    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699fa135_23af_478d_a75d_11518425efd2.slice/crio-6fd5275b086927855ba42d17d9f26867310dee2255bfded76038bc32e00d02c2 WatchSource:0}: Error finding container 6fd5275b086927855ba42d17d9f26867310dee2255bfded76038bc32e00d02c2: Status 404 returned error can't find the container with id 6fd5275b086927855ba42d17d9f26867310dee2255bfded76038bc32e00d02c2
Sep 04 23:58:35.372698 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:35.372639    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf" event={"ID":"699fa135-23af-478d-a75d-11518425efd2","Type":"ContainerStarted","Data":"afe3ff8f3cba22a2a889b4055b7e9c946a0e387ca842d9ee30638d21af16804f"}
Sep 04 23:58:35.373149 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:35.372811    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:35.373149 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:35.372855    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf" event={"ID":"699fa135-23af-478d-a75d-11518425efd2","Type":"ContainerStarted","Data":"6fd5275b086927855ba42d17d9f26867310dee2255bfded76038bc32e00d02c2"}
Sep 04 23:58:35.390486 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:35.390437    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf" podStartSLOduration=3.087388927 podStartE2EDuration="3.390422234s" totalImagesPullingTime="303.033307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:58:35.388386385 +0000 UTC m=+1327.635053475" watchObservedRunningTime="2026-09-04 23:58:35.390422234 +0000 UTC m=+1327.637089313"
Sep 04 23:58:41.383080 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:41.383050    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:58:53.482427 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:53.482392    2562 generic.go:356] "Generic (PLEG): container finished" podID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerID="d8cfea487c9e0309649d1304f1534d355cda89ac16c7ae89ee1ced33d41554e2" exitCode=0
Sep 04 23:58:53.482427 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:53.482432    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp" event={"ID":"59f9eb01-4423-4f9a-9e53-86fc485ddeec","Type":"ContainerDied","Data":"d8cfea487c9e0309649d1304f1534d355cda89ac16c7ae89ee1ced33d41554e2"}
Sep 04 23:58:53.482917 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:53.482833    2562 scope.go:117] "RemoveContainer" containerID="d8cfea487c9e0309649d1304f1534d355cda89ac16c7ae89ee1ced33d41554e2"
Sep 04 23:58:53.483958 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:53.483933    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:58:53.484021 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:53.483996    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:58:53.484060 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:53.484026    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 04 23:58:54.492356 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:54.492324    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp" event={"ID":"59f9eb01-4423-4f9a-9e53-86fc485ddeec","Type":"ContainerStarted","Data":"d6d4869c887e5baa03ca3e05fb1ee4e6d32b3c07381f75990d949783301daf63"}
Sep 04 23:58:54.871776 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:54.871683    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:58:54.871934 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:54.871911    2562 patch_prober.go:28] interesting pod/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 04 23:58:54.871977 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:54.871962    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp" podUID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 04 23:58:55.872260 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:55.872224    2562 patch_prober.go:28] interesting pod/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 04 23:58:55.872640 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:55.872279    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp" podUID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 04 23:58:56.872075 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:56.872034    2562 patch_prober.go:28] interesting pod/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 04 23:58:56.872269 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:56.872094    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp" podUID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 04 23:58:57.872447 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:57.872409    2562 patch_prober.go:28] interesting pod/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 04 23:58:57.872856 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:57.872471    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp" podUID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 04 23:58:58.872426 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:58.872383    2562 patch_prober.go:28] interesting pod/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 04 23:58:58.872633 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:58.872447    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp" podUID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 04 23:58:59.872301 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:59.872266    2562 patch_prober.go:28] interesting pod/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 04 23:58:59.872480 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:58:59.872337    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp" podUID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 04 23:59:00.913594 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:00.913560    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:59:00.914138 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:00.914114    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:59:00.915668 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:00.915639    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:59:14.823202 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:14.823163    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"]
Sep 04 23:59:14.823761 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:14.823423    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm" podUID="6200ed10-cb40-43c9-b9cd-679057437ade" containerName="maas-api" containerID="cri-o://3a1e2e72b228e2e172bcabb0c422b05d7d1dddf9f22db83234ce62572d0f6ca9" gracePeriod=30
Sep 04 23:59:15.083367 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.083304    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:59:15.156174 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.156142    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6200ed10-cb40-43c9-b9cd-679057437ade-openshift-service-ca\") pod \"6200ed10-cb40-43c9-b9cd-679057437ade\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") "
Sep 04 23:59:15.156337 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.156195    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6200ed10-cb40-43c9-b9cd-679057437ade-maas-api-tls\") pod \"6200ed10-cb40-43c9-b9cd-679057437ade\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") "
Sep 04 23:59:15.156337 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.156218    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6200ed10-cb40-43c9-b9cd-679057437ade-metrics-certs\") pod \"6200ed10-cb40-43c9-b9cd-679057437ade\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") "
Sep 04 23:59:15.156337 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.156298    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqhnz\" (UniqueName: \"kubernetes.io/projected/6200ed10-cb40-43c9-b9cd-679057437ade-kube-api-access-mqhnz\") pod \"6200ed10-cb40-43c9-b9cd-679057437ade\" (UID: \"6200ed10-cb40-43c9-b9cd-679057437ade\") "
Sep 04 23:59:15.156512 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.156489    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6200ed10-cb40-43c9-b9cd-679057437ade-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6200ed10-cb40-43c9-b9cd-679057437ade" (UID: "6200ed10-cb40-43c9-b9cd-679057437ade"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:59:15.156750 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.156729    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6200ed10-cb40-43c9-b9cd-679057437ade-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:15.158377 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.158343    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6200ed10-cb40-43c9-b9cd-679057437ade-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6200ed10-cb40-43c9-b9cd-679057437ade" (UID: "6200ed10-cb40-43c9-b9cd-679057437ade"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:59:15.158477 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.158393    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6200ed10-cb40-43c9-b9cd-679057437ade-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6200ed10-cb40-43c9-b9cd-679057437ade" (UID: "6200ed10-cb40-43c9-b9cd-679057437ade"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:59:15.158477 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.158433    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6200ed10-cb40-43c9-b9cd-679057437ade-kube-api-access-mqhnz" (OuterVolumeSpecName: "kube-api-access-mqhnz") pod "6200ed10-cb40-43c9-b9cd-679057437ade" (UID: "6200ed10-cb40-43c9-b9cd-679057437ade"). InnerVolumeSpecName "kube-api-access-mqhnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:59:15.261689 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.258156    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqhnz\" (UniqueName: \"kubernetes.io/projected/6200ed10-cb40-43c9-b9cd-679057437ade-kube-api-access-mqhnz\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:15.261689 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.258196    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6200ed10-cb40-43c9-b9cd-679057437ade-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:15.261689 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.258219    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6200ed10-cb40-43c9-b9cd-679057437ade-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:15.616940 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.616902    2562 generic.go:356] "Generic (PLEG): container finished" podID="6200ed10-cb40-43c9-b9cd-679057437ade" containerID="3a1e2e72b228e2e172bcabb0c422b05d7d1dddf9f22db83234ce62572d0f6ca9" exitCode=0
Sep 04 23:59:15.617144 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.616967    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"
Sep 04 23:59:15.617144 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.616986    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm" event={"ID":"6200ed10-cb40-43c9-b9cd-679057437ade","Type":"ContainerDied","Data":"3a1e2e72b228e2e172bcabb0c422b05d7d1dddf9f22db83234ce62572d0f6ca9"}
Sep 04 23:59:15.617144 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.617044    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm" event={"ID":"6200ed10-cb40-43c9-b9cd-679057437ade","Type":"ContainerDied","Data":"0ad8b90f37fce47fa777db68472ac0f73cdcc4488d8a98156a3562926877f3d3"}
Sep 04 23:59:15.617144 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.617065    2562 scope.go:117] "RemoveContainer" containerID="3a1e2e72b228e2e172bcabb0c422b05d7d1dddf9f22db83234ce62572d0f6ca9"
Sep 04 23:59:15.627017 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.627000    2562 scope.go:117] "RemoveContainer" containerID="3a1e2e72b228e2e172bcabb0c422b05d7d1dddf9f22db83234ce62572d0f6ca9"
Sep 04 23:59:15.627304 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:59:15.627284    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1e2e72b228e2e172bcabb0c422b05d7d1dddf9f22db83234ce62572d0f6ca9\": container with ID starting with 3a1e2e72b228e2e172bcabb0c422b05d7d1dddf9f22db83234ce62572d0f6ca9 not found: ID does not exist" containerID="3a1e2e72b228e2e172bcabb0c422b05d7d1dddf9f22db83234ce62572d0f6ca9"
Sep 04 23:59:15.627349 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.627314    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1e2e72b228e2e172bcabb0c422b05d7d1dddf9f22db83234ce62572d0f6ca9"} err="failed to get container status \"3a1e2e72b228e2e172bcabb0c422b05d7d1dddf9f22db83234ce62572d0f6ca9\": rpc error: code = NotFound desc = could not find container \"3a1e2e72b228e2e172bcabb0c422b05d7d1dddf9f22db83234ce62572d0f6ca9\": container with ID starting with 3a1e2e72b228e2e172bcabb0c422b05d7d1dddf9f22db83234ce62572d0f6ca9 not found: ID does not exist"
Sep 04 23:59:15.638194 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.638165    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"]
Sep 04 23:59:15.641661 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:15.641628    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-87b0d1-865bd8f7b8-n82cm"]
Sep 04 23:59:16.360116 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:16.360080    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6200ed10-cb40-43c9-b9cd-679057437ade" path="/var/lib/kubelet/pods/6200ed10-cb40-43c9-b9cd-679057437ade/volumes"
Sep 04 23:59:37.223561 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.223504    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"]
Sep 04 23:59:37.224135 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.223762    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf" podUID="699fa135-23af-478d-a75d-11518425efd2" containerName="maas-api" containerID="cri-o://afe3ff8f3cba22a2a889b4055b7e9c946a0e387ca842d9ee30638d21af16804f" gracePeriod=30
Sep 04 23:59:37.478565 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.478499    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:59:37.489084 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.489049    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-metrics-certs\") pod \"699fa135-23af-478d-a75d-11518425efd2\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") "
Sep 04 23:59:37.489275 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.489128    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699fa135-23af-478d-a75d-11518425efd2-openshift-service-ca\") pod \"699fa135-23af-478d-a75d-11518425efd2\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") "
Sep 04 23:59:37.489275 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.489190    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cstnr\" (UniqueName: \"kubernetes.io/projected/699fa135-23af-478d-a75d-11518425efd2-kube-api-access-cstnr\") pod \"699fa135-23af-478d-a75d-11518425efd2\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") "
Sep 04 23:59:37.489275 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.489264    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-maas-api-tls\") pod \"699fa135-23af-478d-a75d-11518425efd2\" (UID: \"699fa135-23af-478d-a75d-11518425efd2\") "
Sep 04 23:59:37.489523 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.489502    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699fa135-23af-478d-a75d-11518425efd2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "699fa135-23af-478d-a75d-11518425efd2" (UID: "699fa135-23af-478d-a75d-11518425efd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:59:37.489755 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.489732    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699fa135-23af-478d-a75d-11518425efd2-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:37.492169 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.492132    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "699fa135-23af-478d-a75d-11518425efd2" (UID: "699fa135-23af-478d-a75d-11518425efd2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:59:37.492286 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.492212    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699fa135-23af-478d-a75d-11518425efd2-kube-api-access-cstnr" (OuterVolumeSpecName: "kube-api-access-cstnr") pod "699fa135-23af-478d-a75d-11518425efd2" (UID: "699fa135-23af-478d-a75d-11518425efd2"). InnerVolumeSpecName "kube-api-access-cstnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:59:37.492960 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.492926    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "699fa135-23af-478d-a75d-11518425efd2" (UID: "699fa135-23af-478d-a75d-11518425efd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:59:37.590323 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.590284    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:37.590323 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.590317    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cstnr\" (UniqueName: \"kubernetes.io/projected/699fa135-23af-478d-a75d-11518425efd2-kube-api-access-cstnr\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:37.590323 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.590327    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699fa135-23af-478d-a75d-11518425efd2-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:37.743462 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.743368    2562 generic.go:356] "Generic (PLEG): container finished" podID="699fa135-23af-478d-a75d-11518425efd2" containerID="afe3ff8f3cba22a2a889b4055b7e9c946a0e387ca842d9ee30638d21af16804f" exitCode=0
Sep 04 23:59:37.743462 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.743406    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf" event={"ID":"699fa135-23af-478d-a75d-11518425efd2","Type":"ContainerDied","Data":"afe3ff8f3cba22a2a889b4055b7e9c946a0e387ca842d9ee30638d21af16804f"}
Sep 04 23:59:37.743462 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.743423    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"
Sep 04 23:59:37.743462 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.743451    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf" event={"ID":"699fa135-23af-478d-a75d-11518425efd2","Type":"ContainerDied","Data":"6fd5275b086927855ba42d17d9f26867310dee2255bfded76038bc32e00d02c2"}
Sep 04 23:59:37.743827 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.743472    2562 scope.go:117] "RemoveContainer" containerID="afe3ff8f3cba22a2a889b4055b7e9c946a0e387ca842d9ee30638d21af16804f"
Sep 04 23:59:37.753586 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.753567    2562 scope.go:117] "RemoveContainer" containerID="afe3ff8f3cba22a2a889b4055b7e9c946a0e387ca842d9ee30638d21af16804f"
Sep 04 23:59:37.753925 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:59:37.753893    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe3ff8f3cba22a2a889b4055b7e9c946a0e387ca842d9ee30638d21af16804f\": container with ID starting with afe3ff8f3cba22a2a889b4055b7e9c946a0e387ca842d9ee30638d21af16804f not found: ID does not exist" containerID="afe3ff8f3cba22a2a889b4055b7e9c946a0e387ca842d9ee30638d21af16804f"
Sep 04 23:59:37.754031 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.753929    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe3ff8f3cba22a2a889b4055b7e9c946a0e387ca842d9ee30638d21af16804f"} err="failed to get container status \"afe3ff8f3cba22a2a889b4055b7e9c946a0e387ca842d9ee30638d21af16804f\": rpc error: code = NotFound desc = could not find container \"afe3ff8f3cba22a2a889b4055b7e9c946a0e387ca842d9ee30638d21af16804f\": container with ID starting with afe3ff8f3cba22a2a889b4055b7e9c946a0e387ca842d9ee30638d21af16804f not found: ID does not exist"
Sep 04 23:59:37.765816 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.765788    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"]
Sep 04 23:59:37.771679 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:37.771635    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8cff3b-75b5f46d57-lrbxf"]
Sep 04 23:59:38.358757 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:38.358722    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="699fa135-23af-478d-a75d-11518425efd2" path="/var/lib/kubelet/pods/699fa135-23af-478d-a75d-11518425efd2/volumes"
Sep 04 23:59:43.687774 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:43.687738    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"]
Sep 04 23:59:43.688211 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:43.687950    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp" podUID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerName="istio-proxy" containerID="cri-o://d6d4869c887e5baa03ca3e05fb1ee4e6d32b3c07381f75990d949783301daf63" gracePeriod=30
Sep 04 23:59:45.917474 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:45.917430    2562 patch_prober.go:28] interesting pod/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 04 23:59:45.917873 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:45.917493    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp" podUID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 23:59:48.809766 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:48.809735    2562 generic.go:356] "Generic (PLEG): container finished" podID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerID="d6d4869c887e5baa03ca3e05fb1ee4e6d32b3c07381f75990d949783301daf63" exitCode=0
Sep 04 23:59:48.810167 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:48.809808    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp" event={"ID":"59f9eb01-4423-4f9a-9e53-86fc485ddeec","Type":"ContainerDied","Data":"d6d4869c887e5baa03ca3e05fb1ee4e6d32b3c07381f75990d949783301daf63"}
Sep 04 23:59:48.810167 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:48.809889    2562 scope.go:117] "RemoveContainer" containerID="d8cfea487c9e0309649d1304f1534d355cda89ac16c7ae89ee1ced33d41554e2"
Sep 04 23:59:48.930448 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:48.930422    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:59:49.005337 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.005298    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-workload-certs\") pod \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") "
Sep 04 23:59:49.005337 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.005342    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istiod-ca-cert\") pod \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") "
Sep 04 23:59:49.005585 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.005367    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-data\") pod \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") "
Sep 04 23:59:49.005585 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.005429    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-podinfo\") pod \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") "
Sep 04 23:59:49.005585 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.005524    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-token\") pod \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") "
Sep 04 23:59:49.005585 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.005560    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-envoy\") pod \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") "
Sep 04 23:59:49.005888 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.005586    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-workload-socket\") pod \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") "
Sep 04 23:59:49.005888 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.005597    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "59f9eb01-4423-4f9a-9e53-86fc485ddeec" (UID: "59f9eb01-4423-4f9a-9e53-86fc485ddeec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:59:49.005888 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.005676    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7kjs\" (UniqueName: \"kubernetes.io/projected/59f9eb01-4423-4f9a-9e53-86fc485ddeec-kube-api-access-v7kjs\") pod \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") "
Sep 04 23:59:49.005888 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.005716    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-credential-socket\") pod \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\" (UID: \"59f9eb01-4423-4f9a-9e53-86fc485ddeec\") "
Sep 04 23:59:49.005888 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.005722    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-data" (OuterVolumeSpecName: "istio-data") pod "59f9eb01-4423-4f9a-9e53-86fc485ddeec" (UID: "59f9eb01-4423-4f9a-9e53-86fc485ddeec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:59:49.005888 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.005768    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "59f9eb01-4423-4f9a-9e53-86fc485ddeec" (UID: "59f9eb01-4423-4f9a-9e53-86fc485ddeec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:59:49.006170 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.005939    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "59f9eb01-4423-4f9a-9e53-86fc485ddeec" (UID: "59f9eb01-4423-4f9a-9e53-86fc485ddeec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:59:49.006170 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.006051    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-workload-socket\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:49.006170 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.006071    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-workload-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:49.006170 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.006087    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istiod-ca-cert\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:49.006170 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.006099    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-data\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:49.006170 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.006117    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "59f9eb01-4423-4f9a-9e53-86fc485ddeec" (UID: "59f9eb01-4423-4f9a-9e53-86fc485ddeec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:59:49.008282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.008258    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f9eb01-4423-4f9a-9e53-86fc485ddeec-kube-api-access-v7kjs" (OuterVolumeSpecName: "kube-api-access-v7kjs") pod "59f9eb01-4423-4f9a-9e53-86fc485ddeec" (UID: "59f9eb01-4423-4f9a-9e53-86fc485ddeec"). InnerVolumeSpecName "kube-api-access-v7kjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:59:49.008282 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.008269    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-token" (OuterVolumeSpecName: "istio-token") pod "59f9eb01-4423-4f9a-9e53-86fc485ddeec" (UID: "59f9eb01-4423-4f9a-9e53-86fc485ddeec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:59:49.008459 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.008286    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "59f9eb01-4423-4f9a-9e53-86fc485ddeec" (UID: "59f9eb01-4423-4f9a-9e53-86fc485ddeec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:59:49.008459 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.008289    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "59f9eb01-4423-4f9a-9e53-86fc485ddeec" (UID: "59f9eb01-4423-4f9a-9e53-86fc485ddeec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:59:49.107243 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.107204    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7kjs\" (UniqueName: \"kubernetes.io/projected/59f9eb01-4423-4f9a-9e53-86fc485ddeec-kube-api-access-v7kjs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:49.107243 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.107234    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-credential-socket\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:49.107243 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.107244    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-podinfo\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:49.107243 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.107253    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-token\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:49.107509 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.107261    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f9eb01-4423-4f9a-9e53-86fc485ddeec-istio-envoy\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:49.816504 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.816473    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"
Sep 04 23:59:49.817009 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.816514    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp" event={"ID":"59f9eb01-4423-4f9a-9e53-86fc485ddeec","Type":"ContainerDied","Data":"675875967f3eb2d9a55b6d65d6849ed9ca683aa122e21538be8ada5c0cee1084"}
Sep 04 23:59:49.817009 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.816571    2562 scope.go:117] "RemoveContainer" containerID="d6d4869c887e5baa03ca3e05fb1ee4e6d32b3c07381f75990d949783301daf63"
Sep 04 23:59:49.837391 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.837360    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"]
Sep 04 23:59:49.841247 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:49.841218    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8cff3b-openshift-default-648bfb6d98-rfmhp"]
Sep 04 23:59:50.358873 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:50.358829    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" path="/var/lib/kubelet/pods/59f9eb01-4423-4f9a-9e53-86fc485ddeec/volumes"
Sep 04 23:59:54.380686 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.380625    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"]
Sep 04 23:59:54.381126 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.380901    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8" podUID="40083a10-52ff-4d73-8f77-32ca1ea66111" containerName="maas-api" containerID="cri-o://41dac8f3897fd3315dbd8432f18aeda8bfaec029d036ba6d1fcee7cc82799556" gracePeriod=30
Sep 04 23:59:54.628744 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.628720    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:59:54.761760 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.761727    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q5wp\" (UniqueName: \"kubernetes.io/projected/40083a10-52ff-4d73-8f77-32ca1ea66111-kube-api-access-8q5wp\") pod \"40083a10-52ff-4d73-8f77-32ca1ea66111\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") "
Sep 04 23:59:54.761760 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.761762    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40083a10-52ff-4d73-8f77-32ca1ea66111-openshift-service-ca\") pod \"40083a10-52ff-4d73-8f77-32ca1ea66111\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") "
Sep 04 23:59:54.762028 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.761828    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40083a10-52ff-4d73-8f77-32ca1ea66111-metrics-certs\") pod \"40083a10-52ff-4d73-8f77-32ca1ea66111\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") "
Sep 04 23:59:54.762028 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.761865    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40083a10-52ff-4d73-8f77-32ca1ea66111-maas-api-tls\") pod \"40083a10-52ff-4d73-8f77-32ca1ea66111\" (UID: \"40083a10-52ff-4d73-8f77-32ca1ea66111\") "
Sep 04 23:59:54.762253 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.762227    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40083a10-52ff-4d73-8f77-32ca1ea66111-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40083a10-52ff-4d73-8f77-32ca1ea66111" (UID: "40083a10-52ff-4d73-8f77-32ca1ea66111"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:59:54.764046 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.764020    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40083a10-52ff-4d73-8f77-32ca1ea66111-kube-api-access-8q5wp" (OuterVolumeSpecName: "kube-api-access-8q5wp") pod "40083a10-52ff-4d73-8f77-32ca1ea66111" (UID: "40083a10-52ff-4d73-8f77-32ca1ea66111"). InnerVolumeSpecName "kube-api-access-8q5wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:59:54.764408 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.764391    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40083a10-52ff-4d73-8f77-32ca1ea66111-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "40083a10-52ff-4d73-8f77-32ca1ea66111" (UID: "40083a10-52ff-4d73-8f77-32ca1ea66111"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:59:54.764472 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.764433    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40083a10-52ff-4d73-8f77-32ca1ea66111-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40083a10-52ff-4d73-8f77-32ca1ea66111" (UID: "40083a10-52ff-4d73-8f77-32ca1ea66111"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:59:54.845647 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.845607    2562 generic.go:356] "Generic (PLEG): container finished" podID="40083a10-52ff-4d73-8f77-32ca1ea66111" containerID="41dac8f3897fd3315dbd8432f18aeda8bfaec029d036ba6d1fcee7cc82799556" exitCode=0
Sep 04 23:59:54.845821 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.845676    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"
Sep 04 23:59:54.845821 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.845687    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8" event={"ID":"40083a10-52ff-4d73-8f77-32ca1ea66111","Type":"ContainerDied","Data":"41dac8f3897fd3315dbd8432f18aeda8bfaec029d036ba6d1fcee7cc82799556"}
Sep 04 23:59:54.845821 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.845744    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8" event={"ID":"40083a10-52ff-4d73-8f77-32ca1ea66111","Type":"ContainerDied","Data":"7def5ae806213bb773ee6f7b87c4d1ea9000cdb1ca9746cc78e03379b0ca6733"}
Sep 04 23:59:54.845821 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.845769    2562 scope.go:117] "RemoveContainer" containerID="41dac8f3897fd3315dbd8432f18aeda8bfaec029d036ba6d1fcee7cc82799556"
Sep 04 23:59:54.855546 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.855526    2562 scope.go:117] "RemoveContainer" containerID="41dac8f3897fd3315dbd8432f18aeda8bfaec029d036ba6d1fcee7cc82799556"
Sep 04 23:59:54.855878 ip-10-0-138-66 kubenswrapper[2562]: E0904 23:59:54.855858    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41dac8f3897fd3315dbd8432f18aeda8bfaec029d036ba6d1fcee7cc82799556\": container with ID starting with 41dac8f3897fd3315dbd8432f18aeda8bfaec029d036ba6d1fcee7cc82799556 not found: ID does not exist" containerID="41dac8f3897fd3315dbd8432f18aeda8bfaec029d036ba6d1fcee7cc82799556"
Sep 04 23:59:54.855944 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.855886    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41dac8f3897fd3315dbd8432f18aeda8bfaec029d036ba6d1fcee7cc82799556"} err="failed to get container status \"41dac8f3897fd3315dbd8432f18aeda8bfaec029d036ba6d1fcee7cc82799556\": rpc error: code = NotFound desc = could not find container \"41dac8f3897fd3315dbd8432f18aeda8bfaec029d036ba6d1fcee7cc82799556\": container with ID starting with 41dac8f3897fd3315dbd8432f18aeda8bfaec029d036ba6d1fcee7cc82799556 not found: ID does not exist"
Sep 04 23:59:54.863114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.863090    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q5wp\" (UniqueName: \"kubernetes.io/projected/40083a10-52ff-4d73-8f77-32ca1ea66111-kube-api-access-8q5wp\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:54.863114 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.863112    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40083a10-52ff-4d73-8f77-32ca1ea66111-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:54.863237 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.863122    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40083a10-52ff-4d73-8f77-32ca1ea66111-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:54.863237 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.863131    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40083a10-52ff-4d73-8f77-32ca1ea66111-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 04 23:59:54.867037 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.867012    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"]
Sep 04 23:59:54.870732 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:54.870711    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e35897-86db9bf9d6-t2hn8"]
Sep 04 23:59:56.358715 ip-10-0-138-66 kubenswrapper[2562]: I0904 23:59:56.358640    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40083a10-52ff-4d73-8f77-32ca1ea66111" path="/var/lib/kubelet/pods/40083a10-52ff-4d73-8f77-32ca1ea66111/volumes"
Sep 05 00:00:05.224346 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.224309    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"]
Sep 05 00:00:05.225046 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.225012    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6200ed10-cb40-43c9-b9cd-679057437ade" containerName="maas-api"
Sep 05 00:00:05.225046 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.225040    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6200ed10-cb40-43c9-b9cd-679057437ade" containerName="maas-api"
Sep 05 00:00:05.225180 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.225051    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="699fa135-23af-478d-a75d-11518425efd2" containerName="maas-api"
Sep 05 00:00:05.225180 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.225060    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="699fa135-23af-478d-a75d-11518425efd2" containerName="maas-api"
Sep 05 00:00:05.225180 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.225071    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerName="istio-proxy"
Sep 05 00:00:05.225180 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.225082    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerName="istio-proxy"
Sep 05 00:00:05.225180 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.225098    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerName="istio-proxy"
Sep 05 00:00:05.225180 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.225108    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerName="istio-proxy"
Sep 05 00:00:05.225180 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.225140    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40083a10-52ff-4d73-8f77-32ca1ea66111" containerName="maas-api"
Sep 05 00:00:05.225180 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.225149    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="40083a10-52ff-4d73-8f77-32ca1ea66111" containerName="maas-api"
Sep 05 00:00:05.225424 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.225260    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6200ed10-cb40-43c9-b9cd-679057437ade" containerName="maas-api"
Sep 05 00:00:05.225424 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.225273    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerName="istio-proxy"
Sep 05 00:00:05.225424 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.225284    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f9eb01-4423-4f9a-9e53-86fc485ddeec" containerName="istio-proxy"
Sep 05 00:00:05.225424 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.225294    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="40083a10-52ff-4d73-8f77-32ca1ea66111" containerName="maas-api"
Sep 05 00:00:05.225424 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.225305    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="699fa135-23af-478d-a75d-11518425efd2" containerName="maas-api"
Sep 05 00:00:05.228847 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.228818    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.231437 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.231410    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bea561-openshift-default-dockercfg-f9rn5\""
Sep 05 00:00:05.238130 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.238108    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"]
Sep 05 00:00:05.375963 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.375924    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856fbc29-9b86-47fc-a1de-d7ed37552973-istiod-ca-cert\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.376143 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.375973    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-workload-socket\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.376143 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.376033    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-credential-socket\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.376143 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.376091    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-podinfo\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.376143 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.376119    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-token\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.376302 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.376149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-envoy\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.376302 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.376178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-data\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.376302 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.376215    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-workload-certs\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.376302 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.376282    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krdmz\" (UniqueName: \"kubernetes.io/projected/856fbc29-9b86-47fc-a1de-d7ed37552973-kube-api-access-krdmz\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.477297 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.477197    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-envoy\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.477297 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.477242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-data\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.477525 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.477369    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-workload-certs\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.477525 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.477434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krdmz\" (UniqueName: \"kubernetes.io/projected/856fbc29-9b86-47fc-a1de-d7ed37552973-kube-api-access-krdmz\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.477525 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.477521    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856fbc29-9b86-47fc-a1de-d7ed37552973-istiod-ca-cert\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.477693 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.477562    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-workload-socket\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.477693 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.477580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-data\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.477693 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.477588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-credential-socket\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.477693 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.477642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-podinfo\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.477693 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.477690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-token\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.477957 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.477740    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-workload-certs\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.478047 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.478025    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-workload-socket\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.478165 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.478142    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-credential-socket\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.478317 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.478299    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856fbc29-9b86-47fc-a1de-d7ed37552973-istiod-ca-cert\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.479790 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.479763    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-envoy\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.480039 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.480019    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-podinfo\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.484982 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.484962    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-token\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.485207 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.485182    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krdmz\" (UniqueName: \"kubernetes.io/projected/856fbc29-9b86-47fc-a1de-d7ed37552973-kube-api-access-krdmz\") pod \"e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") " pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.559469 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.559434    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:05.840497 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.840469    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"]
Sep 05 00:00:05.842294 ip-10-0-138-66 kubenswrapper[2562]: W0905 00:00:05.842270    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856fbc29_9b86_47fc_a1de_d7ed37552973.slice/crio-f252583507bd8ad5d23cefd33f9ac3dfa22254a7fb5929280f8fb35503003b83 WatchSource:0}: Error finding container f252583507bd8ad5d23cefd33f9ac3dfa22254a7fb5929280f8fb35503003b83: Status 404 returned error can't find the container with id f252583507bd8ad5d23cefd33f9ac3dfa22254a7fb5929280f8fb35503003b83
Sep 05 00:00:05.844879 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.844849    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 05 00:00:05.844991 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.844920    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 05 00:00:05.844991 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.844949    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 05 00:00:05.909587 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:05.909552    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz" event={"ID":"856fbc29-9b86-47fc-a1de-d7ed37552973","Type":"ContainerStarted","Data":"f252583507bd8ad5d23cefd33f9ac3dfa22254a7fb5929280f8fb35503003b83"}
Sep 05 00:00:06.916544 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:06.916507    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz" event={"ID":"856fbc29-9b86-47fc-a1de-d7ed37552973","Type":"ContainerStarted","Data":"f10615d08ae13a0e04794ca6d8194885d613c1549e21235927c03e0efc3996e5"}
Sep 05 00:00:06.934632 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:06.934581    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz" podStartSLOduration=1.934559937 podStartE2EDuration="1.934559937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 00:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 00:00:06.933460882 +0000 UTC m=+1419.180127973" watchObservedRunningTime="2026-09-05 00:00:06.934559937 +0000 UTC m=+1419.181227016"
Sep 05 00:00:07.560124 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:07.560080    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:07.565376 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:07.565351    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:07.922603 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:07.922523    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:07.923642 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:07.923625    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:09.886545 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:09.886508    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"]
Sep 05 00:00:09.891248 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:09.891220    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:09.893346 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:09.893324    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bea561-serving-cert\""
Sep 05 00:00:09.899288 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:09.899264    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"]
Sep 05 00:00:09.919897 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:09.919865    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:09.920044 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:09.919913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z2rw\" (UniqueName: \"kubernetes.io/projected/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-kube-api-access-6z2rw\") pod \"maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:09.920044 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:09.920011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:09.920150 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:09.920071    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-metrics-certs\") pod \"maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:10.020959 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:10.020928    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:10.021165 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:10.021007    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-metrics-certs\") pod \"maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:10.021165 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:10.021108    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:10.021165 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:10.021141    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z2rw\" (UniqueName: \"kubernetes.io/projected/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-kube-api-access-6z2rw\") pod \"maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:10.021355 ip-10-0-138-66 kubenswrapper[2562]: E0905 00:00:10.021316    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-serving-cert: secret "maas-api-e2e-ipp-a-bea561-serving-cert" not found
Sep 05 00:00:10.021393 ip-10-0-138-66 kubenswrapper[2562]: E0905 00:00:10.021375    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-maas-api-tls podName:a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46 nodeName:}" failed. No retries permitted until 2026-09-05 00:00:10.52135691 +0000 UTC m=+1422.768023994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-maas-api-tls") pod "maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2" (UID: "a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46") : secret "maas-api-e2e-ipp-a-bea561-serving-cert" not found
Sep 05 00:00:10.021701 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:10.021678    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:10.023530 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:10.023508    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-metrics-certs\") pod \"maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:10.043431 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:10.043398    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z2rw\" (UniqueName: \"kubernetes.io/projected/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-kube-api-access-6z2rw\") pod \"maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:10.526928 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:10.526890    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:10.529499 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:10.529468    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:10.805896 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:10.805812    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:11.083999 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.083966    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"]
Sep 05 00:00:11.086831 ip-10-0-138-66 kubenswrapper[2562]: W0905 00:00:11.086794    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda398b5c7_2ea9_49ea_89ff_8e1f83f2fd46.slice/crio-43864a31f2483fd10d146b01fe9b405bc3f93e7efd241a161030d28aac4ca40f WatchSource:0}: Error finding container 43864a31f2483fd10d146b01fe9b405bc3f93e7efd241a161030d28aac4ca40f: Status 404 returned error can't find the container with id 43864a31f2483fd10d146b01fe9b405bc3f93e7efd241a161030d28aac4ca40f
Sep 05 00:00:11.461937 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.461906    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"]
Sep 05 00:00:11.462165 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.462135    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw" podUID="66b62954-f509-4383-8796-21883a065c05" containerName="maas-api" containerID="cri-o://ad60af856720cd8b1d1306173e5734d451039efa8dc1cea6e5f72714f0538b0d" gracePeriod=30
Sep 05 00:00:11.727966 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.727903    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 05 00:00:11.738523 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.738491    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b62954-f509-4383-8796-21883a065c05-maas-api-tls\") pod \"66b62954-f509-4383-8796-21883a065c05\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") "
Sep 05 00:00:11.739297 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.738603    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b62954-f509-4383-8796-21883a065c05-openshift-service-ca\") pod \"66b62954-f509-4383-8796-21883a065c05\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") "
Sep 05 00:00:11.739297 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.738679    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmwj8\" (UniqueName: \"kubernetes.io/projected/66b62954-f509-4383-8796-21883a065c05-kube-api-access-bmwj8\") pod \"66b62954-f509-4383-8796-21883a065c05\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") "
Sep 05 00:00:11.739297 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.738715    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b62954-f509-4383-8796-21883a065c05-metrics-certs\") pod \"66b62954-f509-4383-8796-21883a065c05\" (UID: \"66b62954-f509-4383-8796-21883a065c05\") "
Sep 05 00:00:11.739744 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.739713    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b62954-f509-4383-8796-21883a065c05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66b62954-f509-4383-8796-21883a065c05" (UID: "66b62954-f509-4383-8796-21883a065c05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:00:11.741786 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.741762    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b62954-f509-4383-8796-21883a065c05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66b62954-f509-4383-8796-21883a065c05" (UID: "66b62954-f509-4383-8796-21883a065c05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:00:11.742048 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.742021    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b62954-f509-4383-8796-21883a065c05-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "66b62954-f509-4383-8796-21883a065c05" (UID: "66b62954-f509-4383-8796-21883a065c05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:00:11.742184 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.742155    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b62954-f509-4383-8796-21883a065c05-kube-api-access-bmwj8" (OuterVolumeSpecName: "kube-api-access-bmwj8") pod "66b62954-f509-4383-8796-21883a065c05" (UID: "66b62954-f509-4383-8796-21883a065c05"). InnerVolumeSpecName "kube-api-access-bmwj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:00:11.840268 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.840233    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b62954-f509-4383-8796-21883a065c05-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:00:11.840268 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.840264    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmwj8\" (UniqueName: \"kubernetes.io/projected/66b62954-f509-4383-8796-21883a065c05-kube-api-access-bmwj8\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:00:11.840268 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.840275    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b62954-f509-4383-8796-21883a065c05-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:00:11.840508 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.840283    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b62954-f509-4383-8796-21883a065c05-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:00:11.949372 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.949336    2562 generic.go:356] "Generic (PLEG): container finished" podID="66b62954-f509-4383-8796-21883a065c05" containerID="ad60af856720cd8b1d1306173e5734d451039efa8dc1cea6e5f72714f0538b0d" exitCode=0
Sep 05 00:00:11.949529 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.949384    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"
Sep 05 00:00:11.949529 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.949419    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw" event={"ID":"66b62954-f509-4383-8796-21883a065c05","Type":"ContainerDied","Data":"ad60af856720cd8b1d1306173e5734d451039efa8dc1cea6e5f72714f0538b0d"}
Sep 05 00:00:11.949529 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.949472    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw" event={"ID":"66b62954-f509-4383-8796-21883a065c05","Type":"ContainerDied","Data":"1fe1f726c08abf7120d590ce276979df4a2a61e130d61e08c5b6793bfe01e9bc"}
Sep 05 00:00:11.949529 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.949495    2562 scope.go:117] "RemoveContainer" containerID="ad60af856720cd8b1d1306173e5734d451039efa8dc1cea6e5f72714f0538b0d"
Sep 05 00:00:11.951217 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.951195    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2" event={"ID":"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46","Type":"ContainerStarted","Data":"43864a31f2483fd10d146b01fe9b405bc3f93e7efd241a161030d28aac4ca40f"}
Sep 05 00:00:11.959686 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.959661    2562 scope.go:117] "RemoveContainer" containerID="ad60af856720cd8b1d1306173e5734d451039efa8dc1cea6e5f72714f0538b0d"
Sep 05 00:00:11.960020 ip-10-0-138-66 kubenswrapper[2562]: E0905 00:00:11.959998    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad60af856720cd8b1d1306173e5734d451039efa8dc1cea6e5f72714f0538b0d\": container with ID starting with ad60af856720cd8b1d1306173e5734d451039efa8dc1cea6e5f72714f0538b0d not found: ID does not exist" containerID="ad60af856720cd8b1d1306173e5734d451039efa8dc1cea6e5f72714f0538b0d"
Sep 05 00:00:11.960079 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.960030    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad60af856720cd8b1d1306173e5734d451039efa8dc1cea6e5f72714f0538b0d"} err="failed to get container status \"ad60af856720cd8b1d1306173e5734d451039efa8dc1cea6e5f72714f0538b0d\": rpc error: code = NotFound desc = could not find container \"ad60af856720cd8b1d1306173e5734d451039efa8dc1cea6e5f72714f0538b0d\": container with ID starting with ad60af856720cd8b1d1306173e5734d451039efa8dc1cea6e5f72714f0538b0d not found: ID does not exist"
Sep 05 00:00:11.970628 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.970593    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"]
Sep 05 00:00:11.973636 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:11.973609    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7eb8e-794854f6f9-jgspw"]
Sep 05 00:00:12.360555 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:12.360526    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66b62954-f509-4383-8796-21883a065c05" path="/var/lib/kubelet/pods/66b62954-f509-4383-8796-21883a065c05/volumes"
Sep 05 00:00:13.966394 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:13.966356    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2" event={"ID":"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46","Type":"ContainerStarted","Data":"d27e9baa2278adec2cb097d45696d16e9da76051022c796e8000107650774670"}
Sep 05 00:00:13.966808 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:13.966422    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:13.988362 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:13.988314    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2" podStartSLOduration=2.632372168 podStartE2EDuration="4.988300273s" totalImagesPullingTime="2.355928105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 00:00:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 00:00:13.985900196 +0000 UTC m=+1426.232567276" watchObservedRunningTime="2026-09-05 00:00:13.988300273 +0000 UTC m=+1426.234967352"
Sep 05 00:00:19.976774 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:19.976734    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:00:23.611339 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:23.611158    2562 patch_prober.go:28] interesting pod/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": read tcp 10.132.0.2:35666->10.132.0.71:15021: read: connection reset by peer" start-of-body=
Sep 05 00:00:23.611339 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:23.611276    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz" podUID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": read tcp 10.132.0.2:35666->10.132.0.71:15021: read: connection reset by peer"
Sep 05 00:00:24.021782 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:24.021748    2562 generic.go:356] "Generic (PLEG): container finished" podID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerID="f10615d08ae13a0e04794ca6d8194885d613c1549e21235927c03e0efc3996e5" exitCode=0
Sep 05 00:00:24.021949 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:24.021821    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz" event={"ID":"856fbc29-9b86-47fc-a1de-d7ed37552973","Type":"ContainerDied","Data":"f10615d08ae13a0e04794ca6d8194885d613c1549e21235927c03e0efc3996e5"}
Sep 05 00:00:24.022289 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:24.022274    2562 scope.go:117] "RemoveContainer" containerID="f10615d08ae13a0e04794ca6d8194885d613c1549e21235927c03e0efc3996e5"
Sep 05 00:00:24.023504 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:24.023475    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 05 00:00:24.023563 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:24.023548    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 05 00:00:24.023599 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:24.023585    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 05 00:00:25.034222 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:25.034173    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz" event={"ID":"856fbc29-9b86-47fc-a1de-d7ed37552973","Type":"ContainerStarted","Data":"bd6588532969a9b7dc2d6ee13dabaf32f924f6d4c063a5c5e98b8ec7ced4c594"}
Sep 05 00:00:25.559872 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:25.559830    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:25.560102 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:25.560078    2562 patch_prober.go:28] interesting pod/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 05 00:00:25.560184 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:25.560137    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz" podUID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 05 00:00:26.560804 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:26.560769    2562 patch_prober.go:28] interesting pod/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 05 00:00:26.561237 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:26.560827    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz" podUID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 05 00:00:27.560202 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:27.560165    2562 patch_prober.go:28] interesting pod/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 05 00:00:27.560365 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:27.560217    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz" podUID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 05 00:00:28.560306 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:28.560267    2562 patch_prober.go:28] interesting pod/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 05 00:00:28.560717 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:28.560349    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz" podUID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 05 00:00:29.560988 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:29.560945    2562 patch_prober.go:28] interesting pod/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 05 00:00:29.561399 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:29.561027    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz" podUID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 05 00:00:30.564961 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:30.564919    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:30.565497 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:30.565475    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:00:30.566169 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:00:30.566148    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:01:07.941989 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:01:07.941956    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7nvhm_0b9fe449-ea11-4359-bf6e-e430169a274b/payload-processing/0.log"
Sep 05 00:01:28.304101 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:01:28.304068    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 05 00:01:28.308066 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:01:28.308042    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 05 00:01:28.309570 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:01:28.309549    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 05 00:01:28.312981 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:01:28.312962    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 05 00:01:40.994311 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:01:40.994278    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7nvhm_0b9fe449-ea11-4359-bf6e-e430169a274b/payload-processing/0.log"
Sep 05 00:02:41.477343 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.477311    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"]
Sep 05 00:02:41.477901 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.477561    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2" podUID="a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46" containerName="maas-api" containerID="cri-o://d27e9baa2278adec2cb097d45696d16e9da76051022c796e8000107650774670" gracePeriod=30
Sep 05 00:02:41.735127 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.735054    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:02:41.769441 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.769404    2562 generic.go:356] "Generic (PLEG): container finished" podID="a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46" containerID="d27e9baa2278adec2cb097d45696d16e9da76051022c796e8000107650774670" exitCode=0
Sep 05 00:02:41.769622 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.769452    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"
Sep 05 00:02:41.769622 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.769486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2" event={"ID":"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46","Type":"ContainerDied","Data":"d27e9baa2278adec2cb097d45696d16e9da76051022c796e8000107650774670"}
Sep 05 00:02:41.769622 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.769553    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2" event={"ID":"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46","Type":"ContainerDied","Data":"43864a31f2483fd10d146b01fe9b405bc3f93e7efd241a161030d28aac4ca40f"}
Sep 05 00:02:41.769622 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.769588    2562 scope.go:117] "RemoveContainer" containerID="d27e9baa2278adec2cb097d45696d16e9da76051022c796e8000107650774670"
Sep 05 00:02:41.779908 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.779886    2562 scope.go:117] "RemoveContainer" containerID="d27e9baa2278adec2cb097d45696d16e9da76051022c796e8000107650774670"
Sep 05 00:02:41.780205 ip-10-0-138-66 kubenswrapper[2562]: E0905 00:02:41.780181    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27e9baa2278adec2cb097d45696d16e9da76051022c796e8000107650774670\": container with ID starting with d27e9baa2278adec2cb097d45696d16e9da76051022c796e8000107650774670 not found: ID does not exist" containerID="d27e9baa2278adec2cb097d45696d16e9da76051022c796e8000107650774670"
Sep 05 00:02:41.780279 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.780210    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27e9baa2278adec2cb097d45696d16e9da76051022c796e8000107650774670"} err="failed to get container status \"d27e9baa2278adec2cb097d45696d16e9da76051022c796e8000107650774670\": rpc error: code = NotFound desc = could not find container \"d27e9baa2278adec2cb097d45696d16e9da76051022c796e8000107650774670\": container with ID starting with d27e9baa2278adec2cb097d45696d16e9da76051022c796e8000107650774670 not found: ID does not exist"
Sep 05 00:02:41.840985 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.840951    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z2rw\" (UniqueName: \"kubernetes.io/projected/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-kube-api-access-6z2rw\") pod \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") "
Sep 05 00:02:41.841170 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.841017    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-openshift-service-ca\") pod \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") "
Sep 05 00:02:41.841170 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.841055    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-metrics-certs\") pod \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") "
Sep 05 00:02:41.841170 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.841117    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-maas-api-tls\") pod \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\" (UID: \"a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46\") "
Sep 05 00:02:41.841453 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.841424    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46" (UID: "a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:02:41.841637 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.841574    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:02:41.843243 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.843213    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-kube-api-access-6z2rw" (OuterVolumeSpecName: "kube-api-access-6z2rw") pod "a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46" (UID: "a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46"). InnerVolumeSpecName "kube-api-access-6z2rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:02:41.843629 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.843608    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46" (UID: "a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:02:41.843732 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.843628    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46" (UID: "a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:02:41.942117 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.942083    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z2rw\" (UniqueName: \"kubernetes.io/projected/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-kube-api-access-6z2rw\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:02:41.942117 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.942113    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:02:41.942117 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:41.942121    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:02:42.091909 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:42.091876    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"]
Sep 05 00:02:42.095362 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:42.095331    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bea561-7f689d85c7-lkft2"]
Sep 05 00:02:42.358844 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:02:42.358753    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46" path="/var/lib/kubelet/pods/a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46/volumes"
Sep 05 00:03:29.955149 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:29.955116    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"]
Sep 05 00:03:29.955645 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:29.955389    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz" podUID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerName="istio-proxy" containerID="cri-o://bd6588532969a9b7dc2d6ee13dabaf32f924f6d4c063a5c5e98b8ec7ced4c594" gracePeriod=30
Sep 05 00:03:30.568023 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:30.567988    2562 patch_prober.go:28] interesting pod/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 05 00:03:30.568204 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:30.568056    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz" podUID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 00:03:35.208807 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.208784    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:03:35.358159 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358080    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-envoy\") pod \"856fbc29-9b86-47fc-a1de-d7ed37552973\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") "
Sep 05 00:03:35.358159 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358117    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-workload-socket\") pod \"856fbc29-9b86-47fc-a1de-d7ed37552973\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") "
Sep 05 00:03:35.358159 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358153    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-workload-certs\") pod \"856fbc29-9b86-47fc-a1de-d7ed37552973\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") "
Sep 05 00:03:35.358449 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358178    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-credential-socket\") pod \"856fbc29-9b86-47fc-a1de-d7ed37552973\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") "
Sep 05 00:03:35.358449 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358208    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-data\") pod \"856fbc29-9b86-47fc-a1de-d7ed37552973\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") "
Sep 05 00:03:35.358449 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358228    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krdmz\" (UniqueName: \"kubernetes.io/projected/856fbc29-9b86-47fc-a1de-d7ed37552973-kube-api-access-krdmz\") pod \"856fbc29-9b86-47fc-a1de-d7ed37552973\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") "
Sep 05 00:03:35.358449 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358287    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856fbc29-9b86-47fc-a1de-d7ed37552973-istiod-ca-cert\") pod \"856fbc29-9b86-47fc-a1de-d7ed37552973\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") "
Sep 05 00:03:35.358449 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358323    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-podinfo\") pod \"856fbc29-9b86-47fc-a1de-d7ed37552973\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") "
Sep 05 00:03:35.358449 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358377    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-token\") pod \"856fbc29-9b86-47fc-a1de-d7ed37552973\" (UID: \"856fbc29-9b86-47fc-a1de-d7ed37552973\") "
Sep 05 00:03:35.358809 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358451    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "856fbc29-9b86-47fc-a1de-d7ed37552973" (UID: "856fbc29-9b86-47fc-a1de-d7ed37552973"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:03:35.358809 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358485    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "856fbc29-9b86-47fc-a1de-d7ed37552973" (UID: "856fbc29-9b86-47fc-a1de-d7ed37552973"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:03:35.358809 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358562    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-data" (OuterVolumeSpecName: "istio-data") pod "856fbc29-9b86-47fc-a1de-d7ed37552973" (UID: "856fbc29-9b86-47fc-a1de-d7ed37552973"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:03:35.358809 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358574    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "856fbc29-9b86-47fc-a1de-d7ed37552973" (UID: "856fbc29-9b86-47fc-a1de-d7ed37552973"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:03:35.359007 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358822    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-workload-socket\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:03:35.359007 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358844    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-workload-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:03:35.359007 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358854    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-credential-socket\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:03:35.359007 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358863    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-data\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:03:35.359007 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.358906    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856fbc29-9b86-47fc-a1de-d7ed37552973-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "856fbc29-9b86-47fc-a1de-d7ed37552973" (UID: "856fbc29-9b86-47fc-a1de-d7ed37552973"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:03:35.360600 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.360574    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-token" (OuterVolumeSpecName: "istio-token") pod "856fbc29-9b86-47fc-a1de-d7ed37552973" (UID: "856fbc29-9b86-47fc-a1de-d7ed37552973"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:03:35.360600 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.360588    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "856fbc29-9b86-47fc-a1de-d7ed37552973" (UID: "856fbc29-9b86-47fc-a1de-d7ed37552973"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 00:03:35.360816 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.360741    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856fbc29-9b86-47fc-a1de-d7ed37552973-kube-api-access-krdmz" (OuterVolumeSpecName: "kube-api-access-krdmz") pod "856fbc29-9b86-47fc-a1de-d7ed37552973" (UID: "856fbc29-9b86-47fc-a1de-d7ed37552973"). InnerVolumeSpecName "kube-api-access-krdmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:03:35.360816 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.360749    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "856fbc29-9b86-47fc-a1de-d7ed37552973" (UID: "856fbc29-9b86-47fc-a1de-d7ed37552973"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:03:35.460279 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.460231    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-envoy\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:03:35.460279 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.460275    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krdmz\" (UniqueName: \"kubernetes.io/projected/856fbc29-9b86-47fc-a1de-d7ed37552973-kube-api-access-krdmz\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:03:35.460279 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.460285    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856fbc29-9b86-47fc-a1de-d7ed37552973-istiod-ca-cert\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:03:35.460279 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.460294    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-podinfo\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:03:35.460557 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:35.460304    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856fbc29-9b86-47fc-a1de-d7ed37552973-istio-token\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:03:36.058510 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:36.058469    2562 generic.go:356] "Generic (PLEG): container finished" podID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerID="bd6588532969a9b7dc2d6ee13dabaf32f924f6d4c063a5c5e98b8ec7ced4c594" exitCode=0
Sep 05 00:03:36.058702 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:36.058508    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz" event={"ID":"856fbc29-9b86-47fc-a1de-d7ed37552973","Type":"ContainerDied","Data":"bd6588532969a9b7dc2d6ee13dabaf32f924f6d4c063a5c5e98b8ec7ced4c594"}
Sep 05 00:03:36.058702 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:36.058535    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"
Sep 05 00:03:36.058702 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:36.058567    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz" event={"ID":"856fbc29-9b86-47fc-a1de-d7ed37552973","Type":"ContainerDied","Data":"f252583507bd8ad5d23cefd33f9ac3dfa22254a7fb5929280f8fb35503003b83"}
Sep 05 00:03:36.058702 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:36.058592    2562 scope.go:117] "RemoveContainer" containerID="bd6588532969a9b7dc2d6ee13dabaf32f924f6d4c063a5c5e98b8ec7ced4c594"
Sep 05 00:03:36.067933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:36.067916    2562 scope.go:117] "RemoveContainer" containerID="f10615d08ae13a0e04794ca6d8194885d613c1549e21235927c03e0efc3996e5"
Sep 05 00:03:36.076695 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:36.076498    2562 scope.go:117] "RemoveContainer" containerID="bd6588532969a9b7dc2d6ee13dabaf32f924f6d4c063a5c5e98b8ec7ced4c594"
Sep 05 00:03:36.076804 ip-10-0-138-66 kubenswrapper[2562]: E0905 00:03:36.076784    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6588532969a9b7dc2d6ee13dabaf32f924f6d4c063a5c5e98b8ec7ced4c594\": container with ID starting with bd6588532969a9b7dc2d6ee13dabaf32f924f6d4c063a5c5e98b8ec7ced4c594 not found: ID does not exist" containerID="bd6588532969a9b7dc2d6ee13dabaf32f924f6d4c063a5c5e98b8ec7ced4c594"
Sep 05 00:03:36.076848 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:36.076814    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6588532969a9b7dc2d6ee13dabaf32f924f6d4c063a5c5e98b8ec7ced4c594"} err="failed to get container status \"bd6588532969a9b7dc2d6ee13dabaf32f924f6d4c063a5c5e98b8ec7ced4c594\": rpc error: code = NotFound desc = could not find container \"bd6588532969a9b7dc2d6ee13dabaf32f924f6d4c063a5c5e98b8ec7ced4c594\": container with ID starting with bd6588532969a9b7dc2d6ee13dabaf32f924f6d4c063a5c5e98b8ec7ced4c594 not found: ID does not exist"
Sep 05 00:03:36.076848 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:36.076831    2562 scope.go:117] "RemoveContainer" containerID="f10615d08ae13a0e04794ca6d8194885d613c1549e21235927c03e0efc3996e5"
Sep 05 00:03:36.077077 ip-10-0-138-66 kubenswrapper[2562]: E0905 00:03:36.077059    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10615d08ae13a0e04794ca6d8194885d613c1549e21235927c03e0efc3996e5\": container with ID starting with f10615d08ae13a0e04794ca6d8194885d613c1549e21235927c03e0efc3996e5 not found: ID does not exist" containerID="f10615d08ae13a0e04794ca6d8194885d613c1549e21235927c03e0efc3996e5"
Sep 05 00:03:36.077134 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:36.077087    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10615d08ae13a0e04794ca6d8194885d613c1549e21235927c03e0efc3996e5"} err="failed to get container status \"f10615d08ae13a0e04794ca6d8194885d613c1549e21235927c03e0efc3996e5\": rpc error: code = NotFound desc = could not find container \"f10615d08ae13a0e04794ca6d8194885d613c1549e21235927c03e0efc3996e5\": container with ID starting with f10615d08ae13a0e04794ca6d8194885d613c1549e21235927c03e0efc3996e5 not found: ID does not exist"
Sep 05 00:03:36.080369 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:36.080346    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"]
Sep 05 00:03:36.085713 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:36.085690    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bea561-openshift-default-7785699846-9k2vz"]
Sep 05 00:03:36.358555 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:36.358476    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="856fbc29-9b86-47fc-a1de-d7ed37552973" path="/var/lib/kubelet/pods/856fbc29-9b86-47fc-a1de-d7ed37552973/volumes"
Sep 05 00:03:57.950018 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:57.949954    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"]
Sep 05 00:03:57.950698 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:57.950273    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr" podUID="6f383e85-8df3-4386-90f2-75d57fa6b922" containerName="maas-api" containerID="cri-o://4e5da510b4a11e51f71e9178da18cb7c036038414f5984531335b19a3e2bd4e7" gracePeriod=30
Sep 05 00:03:58.175110 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:58.175069    2562 generic.go:356] "Generic (PLEG): container finished" podID="6f383e85-8df3-4386-90f2-75d57fa6b922" containerID="4e5da510b4a11e51f71e9178da18cb7c036038414f5984531335b19a3e2bd4e7" exitCode=0
Sep 05 00:03:58.175258 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:58.175144    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr" event={"ID":"6f383e85-8df3-4386-90f2-75d57fa6b922","Type":"ContainerDied","Data":"4e5da510b4a11e51f71e9178da18cb7c036038414f5984531335b19a3e2bd4e7"}
Sep 05 00:03:58.200183 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:58.200119    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 05 00:03:58.272832 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:58.272794    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f383e85-8df3-4386-90f2-75d57fa6b922-metrics-certs\") pod \"6f383e85-8df3-4386-90f2-75d57fa6b922\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") "
Sep 05 00:03:58.273024 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:58.272840    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w886\" (UniqueName: \"kubernetes.io/projected/6f383e85-8df3-4386-90f2-75d57fa6b922-kube-api-access-8w886\") pod \"6f383e85-8df3-4386-90f2-75d57fa6b922\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") "
Sep 05 00:03:58.273024 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:58.272922    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f383e85-8df3-4386-90f2-75d57fa6b922-openshift-service-ca\") pod \"6f383e85-8df3-4386-90f2-75d57fa6b922\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") "
Sep 05 00:03:58.273024 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:58.272992    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f383e85-8df3-4386-90f2-75d57fa6b922-maas-api-tls\") pod \"6f383e85-8df3-4386-90f2-75d57fa6b922\" (UID: \"6f383e85-8df3-4386-90f2-75d57fa6b922\") "
Sep 05 00:03:58.273335 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:58.273304    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f383e85-8df3-4386-90f2-75d57fa6b922-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f383e85-8df3-4386-90f2-75d57fa6b922" (UID: "6f383e85-8df3-4386-90f2-75d57fa6b922"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:03:58.275063 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:58.275038    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f383e85-8df3-4386-90f2-75d57fa6b922-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6f383e85-8df3-4386-90f2-75d57fa6b922" (UID: "6f383e85-8df3-4386-90f2-75d57fa6b922"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:03:58.275132 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:58.275058    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f383e85-8df3-4386-90f2-75d57fa6b922-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f383e85-8df3-4386-90f2-75d57fa6b922" (UID: "6f383e85-8df3-4386-90f2-75d57fa6b922"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:03:58.275132 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:58.275040    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f383e85-8df3-4386-90f2-75d57fa6b922-kube-api-access-8w886" (OuterVolumeSpecName: "kube-api-access-8w886") pod "6f383e85-8df3-4386-90f2-75d57fa6b922" (UID: "6f383e85-8df3-4386-90f2-75d57fa6b922"). InnerVolumeSpecName "kube-api-access-8w886". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:03:58.374639 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:58.374612    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f383e85-8df3-4386-90f2-75d57fa6b922-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:03:58.374639 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:58.374637    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f383e85-8df3-4386-90f2-75d57fa6b922-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:03:58.374819 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:58.374646    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f383e85-8df3-4386-90f2-75d57fa6b922-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:03:58.374819 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:58.374684    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w886\" (UniqueName: \"kubernetes.io/projected/6f383e85-8df3-4386-90f2-75d57fa6b922-kube-api-access-8w886\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:03:59.182349 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:59.182317    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"
Sep 05 00:03:59.182349 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:59.182341    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr" event={"ID":"6f383e85-8df3-4386-90f2-75d57fa6b922","Type":"ContainerDied","Data":"101bc2e618a9827a51aa1653000ceb7ed59272f0578d1dfb7458b7d63fee2254"}
Sep 05 00:03:59.182868 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:59.182432    2562 scope.go:117] "RemoveContainer" containerID="4e5da510b4a11e51f71e9178da18cb7c036038414f5984531335b19a3e2bd4e7"
Sep 05 00:03:59.198709 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:59.198681    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"]
Sep 05 00:03:59.204042 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:03:59.204022    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5245c1-5f8798758-qzgwr"]
Sep 05 00:04:00.359314 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:00.359280    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f383e85-8df3-4386-90f2-75d57fa6b922" path="/var/lib/kubelet/pods/6f383e85-8df3-4386-90f2-75d57fa6b922/volumes"
Sep 05 00:04:04.520190 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:04.520155    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"]
Sep 05 00:04:04.520548 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:04.520397    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf" podUID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" containerName="istio-proxy" containerID="cri-o://45ff9fc9cb186140398b1e40961ea2638d32812e91603f2e1e484b8dc60e23c6" gracePeriod=30
Sep 05 00:04:09.786127 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.786064    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 05 00:04:09.887683 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.887628    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istiod-ca-cert\") pod \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") "
Sep 05 00:04:09.887848 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.887699    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-envoy\") pod \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") "
Sep 05 00:04:09.887848 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.887729    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-podinfo\") pod \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") "
Sep 05 00:04:09.887848 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.887765    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-workload-certs\") pod \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") "
Sep 05 00:04:09.887965 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.887915    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvcdk\" (UniqueName: \"kubernetes.io/projected/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-kube-api-access-kvcdk\") pod \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") "
Sep 05 00:04:09.888018 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.887974    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-data\") pod \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") "
Sep 05 00:04:09.888079 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.888023    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-token\") pod \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") "
Sep 05 00:04:09.888137 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.888079    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-credential-socket\") pod \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") "
Sep 05 00:04:09.888137 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.888113    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-workload-socket\") pod \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\" (UID: \"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de\") "
Sep 05 00:04:09.888137 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.888023    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" (UID: "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:04:09.888294 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.888086    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" (UID: "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:04:09.888294 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.888244    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-data" (OuterVolumeSpecName: "istio-data") pod "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" (UID: "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:04:09.888405 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.888350    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" (UID: "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:04:09.888474 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.888451    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" (UID: "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:04:09.888586 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.888569    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istiod-ca-cert\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:09.888682 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.888590    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-workload-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:09.888682 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.888603    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-data\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:09.888682 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.888614    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-credential-socket\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:09.888682 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.888628    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-workload-socket\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:09.890103 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.890073    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" (UID: "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:04:09.890202 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.890119    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" (UID: "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:04:09.890395 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.890369    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" (UID: "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 00:04:09.890482 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.890429    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-kube-api-access-kvcdk" (OuterVolumeSpecName: "kube-api-access-kvcdk") pod "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" (UID: "2ca9520e-1f61-4e0d-bdd8-19f6cfb857de"). InnerVolumeSpecName "kube-api-access-kvcdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:04:09.989895 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.989852    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-token\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:09.989895 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.989886    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-envoy\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:09.989895 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.989898    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-istio-podinfo\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:09.990143 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:09.989910    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvcdk\" (UniqueName: \"kubernetes.io/projected/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de-kube-api-access-kvcdk\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:10.244900 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:10.244868    2562 generic.go:356] "Generic (PLEG): container finished" podID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" containerID="45ff9fc9cb186140398b1e40961ea2638d32812e91603f2e1e484b8dc60e23c6" exitCode=0
Sep 05 00:04:10.245074 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:10.244919    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"
Sep 05 00:04:10.245074 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:10.244929    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf" event={"ID":"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de","Type":"ContainerDied","Data":"45ff9fc9cb186140398b1e40961ea2638d32812e91603f2e1e484b8dc60e23c6"}
Sep 05 00:04:10.245074 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:10.244972    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf" event={"ID":"2ca9520e-1f61-4e0d-bdd8-19f6cfb857de","Type":"ContainerDied","Data":"a135e65e0e761922a7694c9271280c55f3a5a2cde666bb2b9ad56a415060769c"}
Sep 05 00:04:10.245074 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:10.244994    2562 scope.go:117] "RemoveContainer" containerID="45ff9fc9cb186140398b1e40961ea2638d32812e91603f2e1e484b8dc60e23c6"
Sep 05 00:04:10.254843 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:10.254823    2562 scope.go:117] "RemoveContainer" containerID="26ba251087b4b397ad44c4de837187ac8b0fc24e52542773c289e0be4c869045"
Sep 05 00:04:10.263472 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:10.263448    2562 scope.go:117] "RemoveContainer" containerID="45ff9fc9cb186140398b1e40961ea2638d32812e91603f2e1e484b8dc60e23c6"
Sep 05 00:04:10.263861 ip-10-0-138-66 kubenswrapper[2562]: E0905 00:04:10.263839    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ff9fc9cb186140398b1e40961ea2638d32812e91603f2e1e484b8dc60e23c6\": container with ID starting with 45ff9fc9cb186140398b1e40961ea2638d32812e91603f2e1e484b8dc60e23c6 not found: ID does not exist" containerID="45ff9fc9cb186140398b1e40961ea2638d32812e91603f2e1e484b8dc60e23c6"
Sep 05 00:04:10.263954 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:10.263869    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ff9fc9cb186140398b1e40961ea2638d32812e91603f2e1e484b8dc60e23c6"} err="failed to get container status \"45ff9fc9cb186140398b1e40961ea2638d32812e91603f2e1e484b8dc60e23c6\": rpc error: code = NotFound desc = could not find container \"45ff9fc9cb186140398b1e40961ea2638d32812e91603f2e1e484b8dc60e23c6\": container with ID starting with 45ff9fc9cb186140398b1e40961ea2638d32812e91603f2e1e484b8dc60e23c6 not found: ID does not exist"
Sep 05 00:04:10.263954 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:10.263902    2562 scope.go:117] "RemoveContainer" containerID="26ba251087b4b397ad44c4de837187ac8b0fc24e52542773c289e0be4c869045"
Sep 05 00:04:10.264219 ip-10-0-138-66 kubenswrapper[2562]: E0905 00:04:10.264199    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ba251087b4b397ad44c4de837187ac8b0fc24e52542773c289e0be4c869045\": container with ID starting with 26ba251087b4b397ad44c4de837187ac8b0fc24e52542773c289e0be4c869045 not found: ID does not exist" containerID="26ba251087b4b397ad44c4de837187ac8b0fc24e52542773c289e0be4c869045"
Sep 05 00:04:10.264270 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:10.264227    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ba251087b4b397ad44c4de837187ac8b0fc24e52542773c289e0be4c869045"} err="failed to get container status \"26ba251087b4b397ad44c4de837187ac8b0fc24e52542773c289e0be4c869045\": rpc error: code = NotFound desc = could not find container \"26ba251087b4b397ad44c4de837187ac8b0fc24e52542773c289e0be4c869045\": container with ID starting with 26ba251087b4b397ad44c4de837187ac8b0fc24e52542773c289e0be4c869045 not found: ID does not exist"
Sep 05 00:04:10.266931 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:10.266899    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"]
Sep 05 00:04:10.270083 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:10.270063    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5245c1-openshift-default-67df576c88-hhlpf"]
Sep 05 00:04:10.359350 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:10.359311    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" path="/var/lib/kubelet/pods/2ca9520e-1f61-4e0d-bdd8-19f6cfb857de/volumes"
Sep 05 00:04:15.436629 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:15.436582    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"]
Sep 05 00:04:15.437104 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:15.436868    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp" podUID="c1faaa8f-26fc-4058-b7ee-7a5e7356075a" containerName="maas-api" containerID="cri-o://ff55d3c830b8e0d9636e3958d78f2487854826e4c589a0b3f5241aa35e375bc5" gracePeriod=30
Sep 05 00:04:15.688647 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:15.688584    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 05 00:04:15.840643 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:15.840608    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-maas-api-tls\") pod \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") "
Sep 05 00:04:15.840851 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:15.840688    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb8h2\" (UniqueName: \"kubernetes.io/projected/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-kube-api-access-hb8h2\") pod \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") "
Sep 05 00:04:15.840851 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:15.840716    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-metrics-certs\") pod \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") "
Sep 05 00:04:15.840851 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:15.840810    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-openshift-service-ca\") pod \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\" (UID: \"c1faaa8f-26fc-4058-b7ee-7a5e7356075a\") "
Sep 05 00:04:15.841225 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:15.841190    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1faaa8f-26fc-4058-b7ee-7a5e7356075a" (UID: "c1faaa8f-26fc-4058-b7ee-7a5e7356075a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:04:15.842871 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:15.842850    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1faaa8f-26fc-4058-b7ee-7a5e7356075a" (UID: "c1faaa8f-26fc-4058-b7ee-7a5e7356075a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:04:15.842986 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:15.842879    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-kube-api-access-hb8h2" (OuterVolumeSpecName: "kube-api-access-hb8h2") pod "c1faaa8f-26fc-4058-b7ee-7a5e7356075a" (UID: "c1faaa8f-26fc-4058-b7ee-7a5e7356075a"). InnerVolumeSpecName "kube-api-access-hb8h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:04:15.842986 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:15.842942    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c1faaa8f-26fc-4058-b7ee-7a5e7356075a" (UID: "c1faaa8f-26fc-4058-b7ee-7a5e7356075a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:04:15.941873 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:15.941771    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:15.941873 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:15.941806    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:15.941873 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:15.941820    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb8h2\" (UniqueName: \"kubernetes.io/projected/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-kube-api-access-hb8h2\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:15.941873 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:15.941832    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1faaa8f-26fc-4058-b7ee-7a5e7356075a-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:16.278221 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:16.278189    2562 generic.go:356] "Generic (PLEG): container finished" podID="c1faaa8f-26fc-4058-b7ee-7a5e7356075a" containerID="ff55d3c830b8e0d9636e3958d78f2487854826e4c589a0b3f5241aa35e375bc5" exitCode=0
Sep 05 00:04:16.278409 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:16.278239    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"
Sep 05 00:04:16.278409 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:16.278272    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp" event={"ID":"c1faaa8f-26fc-4058-b7ee-7a5e7356075a","Type":"ContainerDied","Data":"ff55d3c830b8e0d9636e3958d78f2487854826e4c589a0b3f5241aa35e375bc5"}
Sep 05 00:04:16.278409 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:16.278319    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp" event={"ID":"c1faaa8f-26fc-4058-b7ee-7a5e7356075a","Type":"ContainerDied","Data":"9c205f4860c621eca46750c18b7fb929f9537c2bb272eaff119029595387aaac"}
Sep 05 00:04:16.278409 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:16.278343    2562 scope.go:117] "RemoveContainer" containerID="ff55d3c830b8e0d9636e3958d78f2487854826e4c589a0b3f5241aa35e375bc5"
Sep 05 00:04:16.287782 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:16.287764    2562 scope.go:117] "RemoveContainer" containerID="ff55d3c830b8e0d9636e3958d78f2487854826e4c589a0b3f5241aa35e375bc5"
Sep 05 00:04:16.288089 ip-10-0-138-66 kubenswrapper[2562]: E0905 00:04:16.288070    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff55d3c830b8e0d9636e3958d78f2487854826e4c589a0b3f5241aa35e375bc5\": container with ID starting with ff55d3c830b8e0d9636e3958d78f2487854826e4c589a0b3f5241aa35e375bc5 not found: ID does not exist" containerID="ff55d3c830b8e0d9636e3958d78f2487854826e4c589a0b3f5241aa35e375bc5"
Sep 05 00:04:16.288157 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:16.288101    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff55d3c830b8e0d9636e3958d78f2487854826e4c589a0b3f5241aa35e375bc5"} err="failed to get container status \"ff55d3c830b8e0d9636e3958d78f2487854826e4c589a0b3f5241aa35e375bc5\": rpc error: code = NotFound desc = could not find container \"ff55d3c830b8e0d9636e3958d78f2487854826e4c589a0b3f5241aa35e375bc5\": container with ID starting with ff55d3c830b8e0d9636e3958d78f2487854826e4c589a0b3f5241aa35e375bc5 not found: ID does not exist"
Sep 05 00:04:16.300505 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:16.300474    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"]
Sep 05 00:04:16.302257 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:16.302235    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-88828d-54bf65d64c-9pmtp"]
Sep 05 00:04:16.359523 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:16.359486    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1faaa8f-26fc-4058-b7ee-7a5e7356075a" path="/var/lib/kubelet/pods/c1faaa8f-26fc-4058-b7ee-7a5e7356075a/volumes"
Sep 05 00:04:24.666101 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:24.666057    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64c89c545-phszt"]
Sep 05 00:04:24.666517 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:24.666308    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-64c89c545-phszt" podUID="12bcdb7b-565e-4bc6-844e-cfb7fcac0d29" containerName="manager" containerID="cri-o://55cf14b47c1d5f719b16027ab7350482e898139146fb6f679855d23781bea22c" gracePeriod=30
Sep 05 00:04:24.919815 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:24.919756    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 05 00:04:25.025757 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.025724    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-webhook-cert\") pod \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\" (UID: \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\") "
Sep 05 00:04:25.025927 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.025850    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46s68\" (UniqueName: \"kubernetes.io/projected/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-kube-api-access-46s68\") pod \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\" (UID: \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\") "
Sep 05 00:04:25.025927 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.025914    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-metrics-certs\") pod \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\" (UID: \"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29\") "
Sep 05 00:04:25.027886 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.027853    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "12bcdb7b-565e-4bc6-844e-cfb7fcac0d29" (UID: "12bcdb7b-565e-4bc6-844e-cfb7fcac0d29"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:04:25.028002 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.027919    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-kube-api-access-46s68" (OuterVolumeSpecName: "kube-api-access-46s68") pod "12bcdb7b-565e-4bc6-844e-cfb7fcac0d29" (UID: "12bcdb7b-565e-4bc6-844e-cfb7fcac0d29"). InnerVolumeSpecName "kube-api-access-46s68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:04:25.028002 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.027940    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "12bcdb7b-565e-4bc6-844e-cfb7fcac0d29" (UID: "12bcdb7b-565e-4bc6-844e-cfb7fcac0d29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:04:25.126623 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.126579    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46s68\" (UniqueName: \"kubernetes.io/projected/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-kube-api-access-46s68\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:25.126623 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.126613    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:25.126623 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.126626    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29-webhook-cert\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:04:25.333107 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.333070    2562 generic.go:356] "Generic (PLEG): container finished" podID="12bcdb7b-565e-4bc6-844e-cfb7fcac0d29" containerID="55cf14b47c1d5f719b16027ab7350482e898139146fb6f679855d23781bea22c" exitCode=0
Sep 05 00:04:25.333320 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.333117    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64c89c545-phszt"
Sep 05 00:04:25.333320 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.333154    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64c89c545-phszt" event={"ID":"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29","Type":"ContainerDied","Data":"55cf14b47c1d5f719b16027ab7350482e898139146fb6f679855d23781bea22c"}
Sep 05 00:04:25.333320 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.333252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64c89c545-phszt" event={"ID":"12bcdb7b-565e-4bc6-844e-cfb7fcac0d29","Type":"ContainerDied","Data":"4127cff23bf1763f383fb774439aaea04a4cfbf2e095739303b84b6a712064b8"}
Sep 05 00:04:25.333320 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.333283    2562 scope.go:117] "RemoveContainer" containerID="55cf14b47c1d5f719b16027ab7350482e898139146fb6f679855d23781bea22c"
Sep 05 00:04:25.344718 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.344700    2562 scope.go:117] "RemoveContainer" containerID="55cf14b47c1d5f719b16027ab7350482e898139146fb6f679855d23781bea22c"
Sep 05 00:04:25.345014 ip-10-0-138-66 kubenswrapper[2562]: E0905 00:04:25.344996    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55cf14b47c1d5f719b16027ab7350482e898139146fb6f679855d23781bea22c\": container with ID starting with 55cf14b47c1d5f719b16027ab7350482e898139146fb6f679855d23781bea22c not found: ID does not exist" containerID="55cf14b47c1d5f719b16027ab7350482e898139146fb6f679855d23781bea22c"
Sep 05 00:04:25.345063 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.345021    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55cf14b47c1d5f719b16027ab7350482e898139146fb6f679855d23781bea22c"} err="failed to get container status \"55cf14b47c1d5f719b16027ab7350482e898139146fb6f679855d23781bea22c\": rpc error: code = NotFound desc = could not find container \"55cf14b47c1d5f719b16027ab7350482e898139146fb6f679855d23781bea22c\": container with ID starting with 55cf14b47c1d5f719b16027ab7350482e898139146fb6f679855d23781bea22c not found: ID does not exist"
Sep 05 00:04:25.356202 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.356183    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64c89c545-phszt"]
Sep 05 00:04:25.359843 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:25.359821    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64c89c545-phszt"]
Sep 05 00:04:26.358532 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:04:26.358496    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12bcdb7b-565e-4bc6-844e-cfb7fcac0d29" path="/var/lib/kubelet/pods/12bcdb7b-565e-4bc6-844e-cfb7fcac0d29/volumes"
Sep 05 00:05:26.184360 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184318    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64c89c545-722vc"]
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184877    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" containerName="istio-proxy"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184896    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" containerName="istio-proxy"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184912    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12bcdb7b-565e-4bc6-844e-cfb7fcac0d29" containerName="manager"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184917    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="12bcdb7b-565e-4bc6-844e-cfb7fcac0d29" containerName="manager"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184924    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" containerName="istio-proxy"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184930    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" containerName="istio-proxy"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184935    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f383e85-8df3-4386-90f2-75d57fa6b922" containerName="maas-api"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184940    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f383e85-8df3-4386-90f2-75d57fa6b922" containerName="maas-api"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184945    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66b62954-f509-4383-8796-21883a065c05" containerName="maas-api"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184950    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="66b62954-f509-4383-8796-21883a065c05" containerName="maas-api"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184955    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46" containerName="maas-api"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184960    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46" containerName="maas-api"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184976    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1faaa8f-26fc-4058-b7ee-7a5e7356075a" containerName="maas-api"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184981    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1faaa8f-26fc-4058-b7ee-7a5e7356075a" containerName="maas-api"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184986    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerName="istio-proxy"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184991    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerName="istio-proxy"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.184998    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerName="istio-proxy"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.185003    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerName="istio-proxy"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.185062    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="12bcdb7b-565e-4bc6-844e-cfb7fcac0d29" containerName="manager"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.185069    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="66b62954-f509-4383-8796-21883a065c05" containerName="maas-api"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.185075    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1faaa8f-26fc-4058-b7ee-7a5e7356075a" containerName="maas-api"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.185082    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerName="istio-proxy"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.185088    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f383e85-8df3-4386-90f2-75d57fa6b922" containerName="maas-api"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.185093    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" containerName="istio-proxy"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.185098    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="856fbc29-9b86-47fc-a1de-d7ed37552973" containerName="istio-proxy"
Sep 05 00:05:26.186933 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.185104    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a398b5c7-2ea9-49ea-89ff-8e1f83f2fd46" containerName="maas-api"
Sep 05 00:05:26.188044 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.188027    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64c89c545-722vc"
Sep 05 00:05:26.191566 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.191537    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 05 00:05:26.191714 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.191588    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-llhsg\""
Sep 05 00:05:26.191714 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.191588    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 05 00:05:26.191714 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.191675    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 05 00:05:26.197709 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.197685    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64c89c545-722vc"]
Sep 05 00:05:26.368639 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.368608    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf569761-5dc3-403a-bfa8-9b3424609cf2-webhook-cert\") pod \"maas-controller-64c89c545-722vc\" (UID: \"bf569761-5dc3-403a-bfa8-9b3424609cf2\") " pod="opendatahub/maas-controller-64c89c545-722vc"
Sep 05 00:05:26.368857 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.368648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xg4g\" (UniqueName: \"kubernetes.io/projected/bf569761-5dc3-403a-bfa8-9b3424609cf2-kube-api-access-6xg4g\") pod \"maas-controller-64c89c545-722vc\" (UID: \"bf569761-5dc3-403a-bfa8-9b3424609cf2\") " pod="opendatahub/maas-controller-64c89c545-722vc"
Sep 05 00:05:26.368857 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.368783    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf569761-5dc3-403a-bfa8-9b3424609cf2-metrics-certs\") pod \"maas-controller-64c89c545-722vc\" (UID: \"bf569761-5dc3-403a-bfa8-9b3424609cf2\") " pod="opendatahub/maas-controller-64c89c545-722vc"
Sep 05 00:05:26.470003 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.469965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xg4g\" (UniqueName: \"kubernetes.io/projected/bf569761-5dc3-403a-bfa8-9b3424609cf2-kube-api-access-6xg4g\") pod \"maas-controller-64c89c545-722vc\" (UID: \"bf569761-5dc3-403a-bfa8-9b3424609cf2\") " pod="opendatahub/maas-controller-64c89c545-722vc"
Sep 05 00:05:26.470190 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.470040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf569761-5dc3-403a-bfa8-9b3424609cf2-metrics-certs\") pod \"maas-controller-64c89c545-722vc\" (UID: \"bf569761-5dc3-403a-bfa8-9b3424609cf2\") " pod="opendatahub/maas-controller-64c89c545-722vc"
Sep 05 00:05:26.470190 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.470129    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf569761-5dc3-403a-bfa8-9b3424609cf2-webhook-cert\") pod \"maas-controller-64c89c545-722vc\" (UID: \"bf569761-5dc3-403a-bfa8-9b3424609cf2\") " pod="opendatahub/maas-controller-64c89c545-722vc"
Sep 05 00:05:26.472507 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.472483    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf569761-5dc3-403a-bfa8-9b3424609cf2-webhook-cert\") pod \"maas-controller-64c89c545-722vc\" (UID: \"bf569761-5dc3-403a-bfa8-9b3424609cf2\") " pod="opendatahub/maas-controller-64c89c545-722vc"
Sep 05 00:05:26.472614 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.472535    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf569761-5dc3-403a-bfa8-9b3424609cf2-metrics-certs\") pod \"maas-controller-64c89c545-722vc\" (UID: \"bf569761-5dc3-403a-bfa8-9b3424609cf2\") " pod="opendatahub/maas-controller-64c89c545-722vc"
Sep 05 00:05:26.477583 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.477566    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xg4g\" (UniqueName: \"kubernetes.io/projected/bf569761-5dc3-403a-bfa8-9b3424609cf2-kube-api-access-6xg4g\") pod \"maas-controller-64c89c545-722vc\" (UID: \"bf569761-5dc3-403a-bfa8-9b3424609cf2\") " pod="opendatahub/maas-controller-64c89c545-722vc"
Sep 05 00:05:26.499545 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.499509    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64c89c545-722vc"
Sep 05 00:05:26.769574 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.769546    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64c89c545-722vc"]
Sep 05 00:05:26.771117 ip-10-0-138-66 kubenswrapper[2562]: W0905 00:05:26.771090    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf569761_5dc3_403a_bfa8_9b3424609cf2.slice/crio-79a36bee697c55d10d6a79cde91bd11bdec5b4f7356e23168fe430334ac649d7 WatchSource:0}: Error finding container 79a36bee697c55d10d6a79cde91bd11bdec5b4f7356e23168fe430334ac649d7: Status 404 returned error can't find the container with id 79a36bee697c55d10d6a79cde91bd11bdec5b4f7356e23168fe430334ac649d7
Sep 05 00:05:26.772435 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:26.772409    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 00:05:27.646221 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:27.646127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64c89c545-722vc" event={"ID":"bf569761-5dc3-403a-bfa8-9b3424609cf2","Type":"ContainerStarted","Data":"ed0b18ef4ed9370de363f9228cd0ec9ae4691e6aec6aa31cef0811cf434e9c5b"}
Sep 05 00:05:27.646221 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:27.646212    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64c89c545-722vc" event={"ID":"bf569761-5dc3-403a-bfa8-9b3424609cf2","Type":"ContainerStarted","Data":"79a36bee697c55d10d6a79cde91bd11bdec5b4f7356e23168fe430334ac649d7"}
Sep 05 00:05:27.646633 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:27.646256    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64c89c545-722vc"
Sep 05 00:05:27.663762 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:27.663713    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-64c89c545-722vc" podStartSLOduration=1.121556049 podStartE2EDuration="1.663695464s" totalImagesPullingTime="542.139415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 00:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 00:05:27.661805404 +0000 UTC m=+1739.908472483" watchObservedRunningTime="2026-09-05 00:05:27.663695464 +0000 UTC m=+1739.910362542"
Sep 05 00:05:38.653350 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:38.653316    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64c89c545-722vc"
Sep 05 00:05:45.924758 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:45.924644    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64c89c545-722vc"]
Sep 05 00:05:45.925268 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:45.924953    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-64c89c545-722vc" podUID="bf569761-5dc3-403a-bfa8-9b3424609cf2" containerName="manager" containerID="cri-o://ed0b18ef4ed9370de363f9228cd0ec9ae4691e6aec6aa31cef0811cf434e9c5b" gracePeriod=30
Sep 05 00:05:46.175323 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.175253    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64c89c545-722vc"
Sep 05 00:05:46.251491 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.251456    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf569761-5dc3-403a-bfa8-9b3424609cf2-metrics-certs\") pod \"bf569761-5dc3-403a-bfa8-9b3424609cf2\" (UID: \"bf569761-5dc3-403a-bfa8-9b3424609cf2\") "
Sep 05 00:05:46.251674 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.251509    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xg4g\" (UniqueName: \"kubernetes.io/projected/bf569761-5dc3-403a-bfa8-9b3424609cf2-kube-api-access-6xg4g\") pod \"bf569761-5dc3-403a-bfa8-9b3424609cf2\" (UID: \"bf569761-5dc3-403a-bfa8-9b3424609cf2\") "
Sep 05 00:05:46.251674 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.251564    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf569761-5dc3-403a-bfa8-9b3424609cf2-webhook-cert\") pod \"bf569761-5dc3-403a-bfa8-9b3424609cf2\" (UID: \"bf569761-5dc3-403a-bfa8-9b3424609cf2\") "
Sep 05 00:05:46.253679 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.253619    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf569761-5dc3-403a-bfa8-9b3424609cf2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bf569761-5dc3-403a-bfa8-9b3424609cf2" (UID: "bf569761-5dc3-403a-bfa8-9b3424609cf2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:05:46.253679 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.253635    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf569761-5dc3-403a-bfa8-9b3424609cf2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bf569761-5dc3-403a-bfa8-9b3424609cf2" (UID: "bf569761-5dc3-403a-bfa8-9b3424609cf2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:05:46.253829 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.253780    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf569761-5dc3-403a-bfa8-9b3424609cf2-kube-api-access-6xg4g" (OuterVolumeSpecName: "kube-api-access-6xg4g") pod "bf569761-5dc3-403a-bfa8-9b3424609cf2" (UID: "bf569761-5dc3-403a-bfa8-9b3424609cf2"). InnerVolumeSpecName "kube-api-access-6xg4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:05:46.352228 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.352194    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf569761-5dc3-403a-bfa8-9b3424609cf2-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:05:46.352228 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.352221    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xg4g\" (UniqueName: \"kubernetes.io/projected/bf569761-5dc3-403a-bfa8-9b3424609cf2-kube-api-access-6xg4g\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:05:46.352228 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.352232    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf569761-5dc3-403a-bfa8-9b3424609cf2-webhook-cert\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:05:46.746616 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.746574    2562 generic.go:356] "Generic (PLEG): container finished" podID="bf569761-5dc3-403a-bfa8-9b3424609cf2" containerID="ed0b18ef4ed9370de363f9228cd0ec9ae4691e6aec6aa31cef0811cf434e9c5b" exitCode=0
Sep 05 00:05:46.746840 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.746630    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64c89c545-722vc"
Sep 05 00:05:46.746840 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.746672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64c89c545-722vc" event={"ID":"bf569761-5dc3-403a-bfa8-9b3424609cf2","Type":"ContainerDied","Data":"ed0b18ef4ed9370de363f9228cd0ec9ae4691e6aec6aa31cef0811cf434e9c5b"}
Sep 05 00:05:46.746840 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.746742    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64c89c545-722vc" event={"ID":"bf569761-5dc3-403a-bfa8-9b3424609cf2","Type":"ContainerDied","Data":"79a36bee697c55d10d6a79cde91bd11bdec5b4f7356e23168fe430334ac649d7"}
Sep 05 00:05:46.746840 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.746771    2562 scope.go:117] "RemoveContainer" containerID="ed0b18ef4ed9370de363f9228cd0ec9ae4691e6aec6aa31cef0811cf434e9c5b"
Sep 05 00:05:46.757953 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.757928    2562 scope.go:117] "RemoveContainer" containerID="ed0b18ef4ed9370de363f9228cd0ec9ae4691e6aec6aa31cef0811cf434e9c5b"
Sep 05 00:05:46.758317 ip-10-0-138-66 kubenswrapper[2562]: E0905 00:05:46.758290    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed0b18ef4ed9370de363f9228cd0ec9ae4691e6aec6aa31cef0811cf434e9c5b\": container with ID starting with ed0b18ef4ed9370de363f9228cd0ec9ae4691e6aec6aa31cef0811cf434e9c5b not found: ID does not exist" containerID="ed0b18ef4ed9370de363f9228cd0ec9ae4691e6aec6aa31cef0811cf434e9c5b"
Sep 05 00:05:46.758408 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.758330    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed0b18ef4ed9370de363f9228cd0ec9ae4691e6aec6aa31cef0811cf434e9c5b"} err="failed to get container status \"ed0b18ef4ed9370de363f9228cd0ec9ae4691e6aec6aa31cef0811cf434e9c5b\": rpc error: code = NotFound desc = could not find container \"ed0b18ef4ed9370de363f9228cd0ec9ae4691e6aec6aa31cef0811cf434e9c5b\": container with ID starting with ed0b18ef4ed9370de363f9228cd0ec9ae4691e6aec6aa31cef0811cf434e9c5b not found: ID does not exist"
Sep 05 00:05:46.764773 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.764749    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64c89c545-722vc"]
Sep 05 00:05:46.769577 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:46.769550    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64c89c545-722vc"]
Sep 05 00:05:47.554599 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.554559    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64c89c545-dw26z"]
Sep 05 00:05:47.555026 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.555012    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf569761-5dc3-403a-bfa8-9b3424609cf2" containerName="manager"
Sep 05 00:05:47.555067 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.555028    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf569761-5dc3-403a-bfa8-9b3424609cf2" containerName="manager"
Sep 05 00:05:47.555122 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.555112    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca9520e-1f61-4e0d-bdd8-19f6cfb857de" containerName="istio-proxy"
Sep 05 00:05:47.555155 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.555122    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf569761-5dc3-403a-bfa8-9b3424609cf2" containerName="manager"
Sep 05 00:05:47.559587 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.559565    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64c89c545-dw26z"
Sep 05 00:05:47.563012 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.562987    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-llhsg\""
Sep 05 00:05:47.563144 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.563020    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 05 00:05:47.563201 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.563138    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 05 00:05:47.563261 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.563204    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 05 00:05:47.567710 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.567687    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64c89c545-dw26z"]
Sep 05 00:05:47.665965 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.665925    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/762eb913-89ba-4212-80e8-1a7e591b2647-webhook-cert\") pod \"maas-controller-64c89c545-dw26z\" (UID: \"762eb913-89ba-4212-80e8-1a7e591b2647\") " pod="opendatahub/maas-controller-64c89c545-dw26z"
Sep 05 00:05:47.666130 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.666040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cms8c\" (UniqueName: \"kubernetes.io/projected/762eb913-89ba-4212-80e8-1a7e591b2647-kube-api-access-cms8c\") pod \"maas-controller-64c89c545-dw26z\" (UID: \"762eb913-89ba-4212-80e8-1a7e591b2647\") " pod="opendatahub/maas-controller-64c89c545-dw26z"
Sep 05 00:05:47.666130 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.666073    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/762eb913-89ba-4212-80e8-1a7e591b2647-metrics-certs\") pod \"maas-controller-64c89c545-dw26z\" (UID: \"762eb913-89ba-4212-80e8-1a7e591b2647\") " pod="opendatahub/maas-controller-64c89c545-dw26z"
Sep 05 00:05:47.766728 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.766689    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/762eb913-89ba-4212-80e8-1a7e591b2647-metrics-certs\") pod \"maas-controller-64c89c545-dw26z\" (UID: \"762eb913-89ba-4212-80e8-1a7e591b2647\") " pod="opendatahub/maas-controller-64c89c545-dw26z"
Sep 05 00:05:47.766918 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.766753    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/762eb913-89ba-4212-80e8-1a7e591b2647-webhook-cert\") pod \"maas-controller-64c89c545-dw26z\" (UID: \"762eb913-89ba-4212-80e8-1a7e591b2647\") " pod="opendatahub/maas-controller-64c89c545-dw26z"
Sep 05 00:05:47.766918 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.766826    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cms8c\" (UniqueName: \"kubernetes.io/projected/762eb913-89ba-4212-80e8-1a7e591b2647-kube-api-access-cms8c\") pod \"maas-controller-64c89c545-dw26z\" (UID: \"762eb913-89ba-4212-80e8-1a7e591b2647\") " pod="opendatahub/maas-controller-64c89c545-dw26z"
Sep 05 00:05:47.769204 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.769174    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/762eb913-89ba-4212-80e8-1a7e591b2647-metrics-certs\") pod \"maas-controller-64c89c545-dw26z\" (UID: \"762eb913-89ba-4212-80e8-1a7e591b2647\") " pod="opendatahub/maas-controller-64c89c545-dw26z"
Sep 05 00:05:47.769314 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.769239    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/762eb913-89ba-4212-80e8-1a7e591b2647-webhook-cert\") pod \"maas-controller-64c89c545-dw26z\" (UID: \"762eb913-89ba-4212-80e8-1a7e591b2647\") " pod="opendatahub/maas-controller-64c89c545-dw26z"
Sep 05 00:05:47.774503 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.774476    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cms8c\" (UniqueName: \"kubernetes.io/projected/762eb913-89ba-4212-80e8-1a7e591b2647-kube-api-access-cms8c\") pod \"maas-controller-64c89c545-dw26z\" (UID: \"762eb913-89ba-4212-80e8-1a7e591b2647\") " pod="opendatahub/maas-controller-64c89c545-dw26z"
Sep 05 00:05:47.871462 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:47.871362    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64c89c545-dw26z"
Sep 05 00:05:48.142283 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:48.142228    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64c89c545-dw26z"]
Sep 05 00:05:48.144627 ip-10-0-138-66 kubenswrapper[2562]: W0905 00:05:48.144596    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762eb913_89ba_4212_80e8_1a7e591b2647.slice/crio-cce276982cf05f1c06b880bac26b95872bef1d61002277006b802620a4eb933a WatchSource:0}: Error finding container cce276982cf05f1c06b880bac26b95872bef1d61002277006b802620a4eb933a: Status 404 returned error can't find the container with id cce276982cf05f1c06b880bac26b95872bef1d61002277006b802620a4eb933a
Sep 05 00:05:48.358279 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:48.358237    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf569761-5dc3-403a-bfa8-9b3424609cf2" path="/var/lib/kubelet/pods/bf569761-5dc3-403a-bfa8-9b3424609cf2/volumes"
Sep 05 00:05:48.758903 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:48.758871    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64c89c545-dw26z" event={"ID":"762eb913-89ba-4212-80e8-1a7e591b2647","Type":"ContainerStarted","Data":"5ba6dca3a729da999791796d32ef58c59cd552384c7d4b67399bd9f7ceea215d"}
Sep 05 00:05:48.759370 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:48.758928    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64c89c545-dw26z" event={"ID":"762eb913-89ba-4212-80e8-1a7e591b2647","Type":"ContainerStarted","Data":"cce276982cf05f1c06b880bac26b95872bef1d61002277006b802620a4eb933a"}
Sep 05 00:05:48.759370 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:48.758997    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64c89c545-dw26z"
Sep 05 00:05:48.777636 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:48.777591    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-64c89c545-dw26z" podStartSLOduration=1.367295515 podStartE2EDuration="1.777575659s" totalImagesPullingTime="410.280144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 00:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 00:05:48.774619444 +0000 UTC m=+1761.021286526" watchObservedRunningTime="2026-09-05 00:05:48.777575659 +0000 UTC m=+1761.024242737"
Sep 05 00:05:59.769905 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:05:59.769873    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64c89c545-dw26z"
Sep 05 00:06:28.341879 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:06:28.341837    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 05 00:06:28.346916 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:06:28.346890    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 05 00:06:28.348571 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:06:28.348550    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 05 00:06:28.360068 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:06:28.360044    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d75ld_3e04aab2-c27c-4ba1-b1df-3d2daa4f13ee/ovn-acl-logging/0.log"
Sep 05 00:09:43.882061 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:43.882029    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57989d57b9-5887r"]
Sep 05 00:09:43.885408 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:43.885383    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:43.887781 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:43.887759    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mnd5p\""
Sep 05 00:09:43.897830 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:43.897806    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57989d57b9-5887r"]
Sep 05 00:09:43.956877 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:43.956848    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6brhk\" (UniqueName: \"kubernetes.io/projected/09da0bc6-9ef0-4050-9522-d5c48fe61c97-kube-api-access-6brhk\") pod \"maas-api-57989d57b9-5887r\" (UID: \"09da0bc6-9ef0-4050-9522-d5c48fe61c97\") " pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:43.957046 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:43.956902    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09da0bc6-9ef0-4050-9522-d5c48fe61c97-openshift-service-ca\") pod \"maas-api-57989d57b9-5887r\" (UID: \"09da0bc6-9ef0-4050-9522-d5c48fe61c97\") " pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:43.957046 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:43.956949    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09da0bc6-9ef0-4050-9522-d5c48fe61c97-maas-api-tls\") pod \"maas-api-57989d57b9-5887r\" (UID: \"09da0bc6-9ef0-4050-9522-d5c48fe61c97\") " pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:43.957046 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:43.957030    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09da0bc6-9ef0-4050-9522-d5c48fe61c97-metrics-certs\") pod \"maas-api-57989d57b9-5887r\" (UID: \"09da0bc6-9ef0-4050-9522-d5c48fe61c97\") " pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:44.057951 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:44.057916    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09da0bc6-9ef0-4050-9522-d5c48fe61c97-openshift-service-ca\") pod \"maas-api-57989d57b9-5887r\" (UID: \"09da0bc6-9ef0-4050-9522-d5c48fe61c97\") " pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:44.057951 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:44.057952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09da0bc6-9ef0-4050-9522-d5c48fe61c97-maas-api-tls\") pod \"maas-api-57989d57b9-5887r\" (UID: \"09da0bc6-9ef0-4050-9522-d5c48fe61c97\") " pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:44.058152 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:44.058001    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09da0bc6-9ef0-4050-9522-d5c48fe61c97-metrics-certs\") pod \"maas-api-57989d57b9-5887r\" (UID: \"09da0bc6-9ef0-4050-9522-d5c48fe61c97\") " pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:44.058152 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:44.058037    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6brhk\" (UniqueName: \"kubernetes.io/projected/09da0bc6-9ef0-4050-9522-d5c48fe61c97-kube-api-access-6brhk\") pod \"maas-api-57989d57b9-5887r\" (UID: \"09da0bc6-9ef0-4050-9522-d5c48fe61c97\") " pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:44.058718 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:44.058689    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09da0bc6-9ef0-4050-9522-d5c48fe61c97-openshift-service-ca\") pod \"maas-api-57989d57b9-5887r\" (UID: \"09da0bc6-9ef0-4050-9522-d5c48fe61c97\") " pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:44.060559 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:44.060538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09da0bc6-9ef0-4050-9522-d5c48fe61c97-metrics-certs\") pod \"maas-api-57989d57b9-5887r\" (UID: \"09da0bc6-9ef0-4050-9522-d5c48fe61c97\") " pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:44.060684 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:44.060587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09da0bc6-9ef0-4050-9522-d5c48fe61c97-maas-api-tls\") pod \"maas-api-57989d57b9-5887r\" (UID: \"09da0bc6-9ef0-4050-9522-d5c48fe61c97\") " pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:44.065529 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:44.065505    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6brhk\" (UniqueName: \"kubernetes.io/projected/09da0bc6-9ef0-4050-9522-d5c48fe61c97-kube-api-access-6brhk\") pod \"maas-api-57989d57b9-5887r\" (UID: \"09da0bc6-9ef0-4050-9522-d5c48fe61c97\") " pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:44.196972 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:44.196878    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:44.469836 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:44.469809    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57989d57b9-5887r"]
Sep 05 00:09:44.472169 ip-10-0-138-66 kubenswrapper[2562]: W0905 00:09:44.472142    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09da0bc6_9ef0_4050_9522_d5c48fe61c97.slice/crio-a92efed672bb6925aeb52be1ec7406d19466eb7bb726ebe242f4c02d08aafff8 WatchSource:0}: Error finding container a92efed672bb6925aeb52be1ec7406d19466eb7bb726ebe242f4c02d08aafff8: Status 404 returned error can't find the container with id a92efed672bb6925aeb52be1ec7406d19466eb7bb726ebe242f4c02d08aafff8
Sep 05 00:09:44.956140 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:44.956108    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r" event={"ID":"09da0bc6-9ef0-4050-9522-d5c48fe61c97","Type":"ContainerStarted","Data":"a92efed672bb6925aeb52be1ec7406d19466eb7bb726ebe242f4c02d08aafff8"}
Sep 05 00:09:45.962603 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:45.962563    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r" event={"ID":"09da0bc6-9ef0-4050-9522-d5c48fe61c97","Type":"ContainerStarted","Data":"cc4c620ff1173835a723d6bdd7ed3b6d91a059069e8752d2f2db124ef2c66de4"}
Sep 05 00:09:45.963156 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:45.962698    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:45.979088 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:45.979025    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r" podStartSLOduration=2.536346709 podStartE2EDuration="2.979004494s" totalImagesPullingTime="442.657785ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 00:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 00:09:45.978302017 +0000 UTC m=+1998.224969097" watchObservedRunningTime="2026-09-05 00:09:45.979004494 +0000 UTC m=+1998.225671573"
Sep 05 00:09:51.973037 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:51.973009    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57989d57b9-5887r"
Sep 05 00:09:52.021501 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:52.021461    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"]
Sep 05 00:09:52.021762 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:52.021728    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr" podUID="499191ea-f914-491c-9c86-786bac33e8cc" containerName="maas-api" containerID="cri-o://7125c0f8d8ff436d3bb5852fce011a28f7fa25f1ac4cffe90c237703e24a0e16" gracePeriod=30
Sep 05 00:09:53.272800 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:53.272772    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 05 00:09:53.342064 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:53.341980    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/499191ea-f914-491c-9c86-786bac33e8cc-metrics-certs\") pod \"499191ea-f914-491c-9c86-786bac33e8cc\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") "
Sep 05 00:09:53.342221 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:53.342110    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/499191ea-f914-491c-9c86-786bac33e8cc-openshift-service-ca\") pod \"499191ea-f914-491c-9c86-786bac33e8cc\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") "
Sep 05 00:09:53.342221 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:53.342132    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/499191ea-f914-491c-9c86-786bac33e8cc-maas-api-tls\") pod \"499191ea-f914-491c-9c86-786bac33e8cc\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") "
Sep 05 00:09:53.342221 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:53.342150    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjzlb\" (UniqueName: \"kubernetes.io/projected/499191ea-f914-491c-9c86-786bac33e8cc-kube-api-access-sjzlb\") pod \"499191ea-f914-491c-9c86-786bac33e8cc\" (UID: \"499191ea-f914-491c-9c86-786bac33e8cc\") "
Sep 05 00:09:53.342553 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:53.342511    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/499191ea-f914-491c-9c86-786bac33e8cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "499191ea-f914-491c-9c86-786bac33e8cc" (UID: "499191ea-f914-491c-9c86-786bac33e8cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:09:53.344266 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:53.344236    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/499191ea-f914-491c-9c86-786bac33e8cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "499191ea-f914-491c-9c86-786bac33e8cc" (UID: "499191ea-f914-491c-9c86-786bac33e8cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:09:53.344350 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:53.344272    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/499191ea-f914-491c-9c86-786bac33e8cc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "499191ea-f914-491c-9c86-786bac33e8cc" (UID: "499191ea-f914-491c-9c86-786bac33e8cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:09:53.344350 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:53.344284    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499191ea-f914-491c-9c86-786bac33e8cc-kube-api-access-sjzlb" (OuterVolumeSpecName: "kube-api-access-sjzlb") pod "499191ea-f914-491c-9c86-786bac33e8cc" (UID: "499191ea-f914-491c-9c86-786bac33e8cc"). InnerVolumeSpecName "kube-api-access-sjzlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:09:53.443920 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:53.443887    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/499191ea-f914-491c-9c86-786bac33e8cc-openshift-service-ca\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:09:53.443920 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:53.443915    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/499191ea-f914-491c-9c86-786bac33e8cc-maas-api-tls\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:09:53.443920 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:53.443924    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjzlb\" (UniqueName: \"kubernetes.io/projected/499191ea-f914-491c-9c86-786bac33e8cc-kube-api-access-sjzlb\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:09:53.444152 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:53.443933    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/499191ea-f914-491c-9c86-786bac33e8cc-metrics-certs\") on node \"ip-10-0-138-66.ec2.internal\" DevicePath \"\""
Sep 05 00:09:54.003840 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:54.003806    2562 generic.go:356] "Generic (PLEG): container finished" podID="499191ea-f914-491c-9c86-786bac33e8cc" containerID="7125c0f8d8ff436d3bb5852fce011a28f7fa25f1ac4cffe90c237703e24a0e16" exitCode=0
Sep 05 00:09:54.004015 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:54.003861    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"
Sep 05 00:09:54.004015 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:54.003890    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr" event={"ID":"499191ea-f914-491c-9c86-786bac33e8cc","Type":"ContainerDied","Data":"7125c0f8d8ff436d3bb5852fce011a28f7fa25f1ac4cffe90c237703e24a0e16"}
Sep 05 00:09:54.004015 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:54.003976    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr" event={"ID":"499191ea-f914-491c-9c86-786bac33e8cc","Type":"ContainerDied","Data":"ee267e549a0c46da55dd1ca5bbdeaa37684774208a24ee8cdb8c48f645dbce5c"}
Sep 05 00:09:54.004015 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:54.004004    2562 scope.go:117] "RemoveContainer" containerID="7125c0f8d8ff436d3bb5852fce011a28f7fa25f1ac4cffe90c237703e24a0e16"
Sep 05 00:09:54.013552 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:54.013535    2562 scope.go:117] "RemoveContainer" containerID="7125c0f8d8ff436d3bb5852fce011a28f7fa25f1ac4cffe90c237703e24a0e16"
Sep 05 00:09:54.013868 ip-10-0-138-66 kubenswrapper[2562]: E0905 00:09:54.013849    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7125c0f8d8ff436d3bb5852fce011a28f7fa25f1ac4cffe90c237703e24a0e16\": container with ID starting with 7125c0f8d8ff436d3bb5852fce011a28f7fa25f1ac4cffe90c237703e24a0e16 not found: ID does not exist" containerID="7125c0f8d8ff436d3bb5852fce011a28f7fa25f1ac4cffe90c237703e24a0e16"
Sep 05 00:09:54.013929 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:54.013878    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7125c0f8d8ff436d3bb5852fce011a28f7fa25f1ac4cffe90c237703e24a0e16"} err="failed to get container status \"7125c0f8d8ff436d3bb5852fce011a28f7fa25f1ac4cffe90c237703e24a0e16\": rpc error: code = NotFound desc = could not find container \"7125c0f8d8ff436d3bb5852fce011a28f7fa25f1ac4cffe90c237703e24a0e16\": container with ID starting with 7125c0f8d8ff436d3bb5852fce011a28f7fa25f1ac4cffe90c237703e24a0e16 not found: ID does not exist"
Sep 05 00:09:54.024977 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:54.024952    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"]
Sep 05 00:09:54.030044 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:54.030021    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-66b67c778f-lvtdr"]
Sep 05 00:09:54.357963 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:09:54.357884    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="499191ea-f914-491c-9c86-786bac33e8cc" path="/var/lib/kubelet/pods/499191ea-f914-491c-9c86-786bac33e8cc/volumes"
Sep 05 00:10:24.104471 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:24.104436    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-57t66_f180203c-534c-4b4a-bc01-fb6885e26f22/manager/0.log"
Sep 05 00:10:24.352684 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:24.351975    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64c89c545-dw26z_762eb913-89ba-4212-80e8-1a7e591b2647/manager/0.log"
Sep 05 00:10:24.709998 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:24.709966    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b9496bc86-2xc4x_f1c5f4cf-5b1a-4aad-b528-592dcb82370f/copy-manifests/0.log"
Sep 05 00:10:24.716816 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:24.716793    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b9496bc86-2xc4x_f1c5f4cf-5b1a-4aad-b528-592dcb82370f/manager/0.log"
Sep 05 00:10:25.294374 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:25.294341    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57989d57b9-5887r_09da0bc6-9ef0-4050-9522-d5c48fe61c97/maas-api/0.log"
Sep 05 00:10:25.600790 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:25.600707    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-wth2h_15ebf5ad-5286-4ba7-b5e2-bf78192b5091/postgres/0.log"
Sep 05 00:10:26.468470 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:26.468435    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-zggq7_74e16c05-c1ed-4af2-92f7-d7c85f6d8ce2/manager/0.log"
Sep 05 00:10:26.587060 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:26.587034    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-gl4w2_cf265509-3e29-4c30-aee7-28df1e5912b9/manager/0.log"
Sep 05 00:10:26.698780 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:26.698756    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-rtj8c_474c9c83-b3b1-43f2-9863-3b8be53cdd11/kuadrant-console-plugin/0.log"
Sep 05 00:10:27.038049 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:27.038014    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-m4dvc_632a7bdc-5cb2-453c-87fb-72557a347170/manager/0.log"
Sep 05 00:10:27.624009 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:27.623978    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d989d9b5-46mnh_c8007e7f-c503-42f0-a3b5-fdd289603da3/kube-auth-proxy/0.log"
Sep 05 00:10:27.853693 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:27.853628    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-ln8g4_d9907f4b-6c2c-4b46-b927-ff56ad289e45/istio-proxy/1.log"
Sep 05 00:10:28.086565 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:28.086533    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7nvhm_0b9fe449-ea11-4359-bf6e-e430169a274b/payload-processing/0.log"
Sep 05 00:10:28.196132 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:28.196102    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76b88fd7dd-5knhq_1b11f31d-54cf-49b7-a4b7-ae4984221a56/router/0.log"
Sep 05 00:10:28.527514 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:28.527481    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-6x694_35f164f5-e979-4dbc-b894-ff0681e0e758/main/0.log"
Sep 05 00:10:28.746882 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:28.746855    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-vbmkf_7a4a6b81-c17e-45e9-b262-2a1b8da54710/main/0.log"
Sep 05 00:10:29.078080 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:29.078051    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-qc2fl_f4ddf861-9ca0-4d4d-a3e8-ace18e9f48ee/main/0.log"
Sep 05 00:10:37.670043 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:37.670009    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hq8l5_e7234a20-6651-4386-8521-484e928babba/global-pull-secret-syncer/0.log"
Sep 05 00:10:37.878431 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:37.878397    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nc25g_48d8ba50-68de-4777-9e62-b7022698a5ca/konnectivity-agent/0.log"
Sep 05 00:10:37.963372 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:37.963339    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-66.ec2.internal_34cc8fecf92fb93bf9c3003de27a7737/haproxy/0.log"
Sep 05 00:10:42.773777 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:42.773719    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-zggq7_74e16c05-c1ed-4af2-92f7-d7c85f6d8ce2/manager/0.log"
Sep 05 00:10:42.795809 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:42.795764    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-gl4w2_cf265509-3e29-4c30-aee7-28df1e5912b9/manager/0.log"
Sep 05 00:10:42.817769 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:42.817744    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-rtj8c_474c9c83-b3b1-43f2-9863-3b8be53cdd11/kuadrant-console-plugin/0.log"
Sep 05 00:10:42.973744 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:42.973707    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-m4dvc_632a7bdc-5cb2-453c-87fb-72557a347170/manager/0.log"
Sep 05 00:10:44.511754 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:44.511714    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-vd6n9_08ef1223-76e2-4834-af84-b9ca9a19b579/cluster-monitoring-operator/0.log"
Sep 05 00:10:44.674238 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:44.674200    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5b8gj_dc21b587-b979-4a80-bd88-7e7b40dbacad/node-exporter/0.log"
Sep 05 00:10:44.696545 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:44.696512    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5b8gj_dc21b587-b979-4a80-bd88-7e7b40dbacad/kube-rbac-proxy/0.log"
Sep 05 00:10:44.715524 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:44.715497    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5b8gj_dc21b587-b979-4a80-bd88-7e7b40dbacad/init-textfile/0.log"
Sep 05 00:10:46.747097 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:46.747065    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-6fj75_3fa30820-fda2-4e06-80a1-6e9de51833ea/networking-console-plugin/0.log"
Sep 05 00:10:47.328567 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.328536    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/1.log"
Sep 05 00:10:47.337769 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.337745    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5c7r9_8c3a8a40-d2d2-4093-85dc-25b176df91d2/console-operator/2.log"
Sep 05 00:10:47.531331 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.531298    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"]
Sep 05 00:10:47.531729 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.531716    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="499191ea-f914-491c-9c86-786bac33e8cc" containerName="maas-api"
Sep 05 00:10:47.531778 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.531731    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="499191ea-f914-491c-9c86-786bac33e8cc" containerName="maas-api"
Sep 05 00:10:47.531814 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.531806    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="499191ea-f914-491c-9c86-786bac33e8cc" containerName="maas-api"
Sep 05 00:10:47.533897 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.533878    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.536347 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.536320    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c2q7w\"/\"openshift-service-ca.crt\""
Sep 05 00:10:47.536485 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.536437    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c2q7w\"/\"kube-root-ca.crt\""
Sep 05 00:10:47.536573 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.536557    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c2q7w\"/\"default-dockercfg-8v5v9\""
Sep 05 00:10:47.545063 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.545041    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"]
Sep 05 00:10:47.635236 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.635157    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c511daba-4ef8-46c5-bc1f-3a97a03a0331-lib-modules\") pod \"perf-node-gather-daemonset-9j8gl\" (UID: \"c511daba-4ef8-46c5-bc1f-3a97a03a0331\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.635236 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.635197    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c511daba-4ef8-46c5-bc1f-3a97a03a0331-sys\") pod \"perf-node-gather-daemonset-9j8gl\" (UID: \"c511daba-4ef8-46c5-bc1f-3a97a03a0331\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.635236 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.635212    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c511daba-4ef8-46c5-bc1f-3a97a03a0331-proc\") pod \"perf-node-gather-daemonset-9j8gl\" (UID: \"c511daba-4ef8-46c5-bc1f-3a97a03a0331\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.635449 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.635306    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c511daba-4ef8-46c5-bc1f-3a97a03a0331-podres\") pod \"perf-node-gather-daemonset-9j8gl\" (UID: \"c511daba-4ef8-46c5-bc1f-3a97a03a0331\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.635449 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.635374    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67vqg\" (UniqueName: \"kubernetes.io/projected/c511daba-4ef8-46c5-bc1f-3a97a03a0331-kube-api-access-67vqg\") pod \"perf-node-gather-daemonset-9j8gl\" (UID: \"c511daba-4ef8-46c5-bc1f-3a97a03a0331\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.736153 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.736114    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67vqg\" (UniqueName: \"kubernetes.io/projected/c511daba-4ef8-46c5-bc1f-3a97a03a0331-kube-api-access-67vqg\") pod \"perf-node-gather-daemonset-9j8gl\" (UID: \"c511daba-4ef8-46c5-bc1f-3a97a03a0331\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.736328 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.736161    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c511daba-4ef8-46c5-bc1f-3a97a03a0331-lib-modules\") pod \"perf-node-gather-daemonset-9j8gl\" (UID: \"c511daba-4ef8-46c5-bc1f-3a97a03a0331\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.736328 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.736199    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c511daba-4ef8-46c5-bc1f-3a97a03a0331-sys\") pod \"perf-node-gather-daemonset-9j8gl\" (UID: \"c511daba-4ef8-46c5-bc1f-3a97a03a0331\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.736328 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.736215    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c511daba-4ef8-46c5-bc1f-3a97a03a0331-proc\") pod \"perf-node-gather-daemonset-9j8gl\" (UID: \"c511daba-4ef8-46c5-bc1f-3a97a03a0331\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.736328 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.736276    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c511daba-4ef8-46c5-bc1f-3a97a03a0331-podres\") pod \"perf-node-gather-daemonset-9j8gl\" (UID: \"c511daba-4ef8-46c5-bc1f-3a97a03a0331\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.736506 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.736351    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c511daba-4ef8-46c5-bc1f-3a97a03a0331-proc\") pod \"perf-node-gather-daemonset-9j8gl\" (UID: \"c511daba-4ef8-46c5-bc1f-3a97a03a0331\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.736506 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.736351    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c511daba-4ef8-46c5-bc1f-3a97a03a0331-sys\") pod \"perf-node-gather-daemonset-9j8gl\" (UID: \"c511daba-4ef8-46c5-bc1f-3a97a03a0331\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.736506 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.736392    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c511daba-4ef8-46c5-bc1f-3a97a03a0331-lib-modules\") pod \"perf-node-gather-daemonset-9j8gl\" (UID: \"c511daba-4ef8-46c5-bc1f-3a97a03a0331\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.736506 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.736411    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c511daba-4ef8-46c5-bc1f-3a97a03a0331-podres\") pod \"perf-node-gather-daemonset-9j8gl\" (UID: \"c511daba-4ef8-46c5-bc1f-3a97a03a0331\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.743879 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.743846    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67vqg\" (UniqueName: \"kubernetes.io/projected/c511daba-4ef8-46c5-bc1f-3a97a03a0331-kube-api-access-67vqg\") pod \"perf-node-gather-daemonset-9j8gl\" (UID: \"c511daba-4ef8-46c5-bc1f-3a97a03a0331\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.844946 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.844913    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:47.901539 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:47.901507    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-td8gm_082f581e-6378-4981-bba8-fd66fa8923e1/download-server/0.log"
Sep 05 00:10:48.120727 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:48.120699    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"]
Sep 05 00:10:48.122847 ip-10-0-138-66 kubenswrapper[2562]: W0905 00:10:48.122800    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc511daba_4ef8_46c5_bc1f_3a97a03a0331.slice/crio-fe187294d6e0e60f07dac7bf93bfb5342a34f0b420cf634a4a1f5cba726266a8 WatchSource:0}: Error finding container fe187294d6e0e60f07dac7bf93bfb5342a34f0b420cf634a4a1f5cba726266a8: Status 404 returned error can't find the container with id fe187294d6e0e60f07dac7bf93bfb5342a34f0b420cf634a4a1f5cba726266a8
Sep 05 00:10:48.289842 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:48.289804    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl" event={"ID":"c511daba-4ef8-46c5-bc1f-3a97a03a0331","Type":"ContainerStarted","Data":"fc40d34ea36a7b8f8aa942d8b2644de0a5774007ca1d30ff9eaa9cc8046939c7"}
Sep 05 00:10:48.289997 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:48.289859    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl" event={"ID":"c511daba-4ef8-46c5-bc1f-3a97a03a0331","Type":"ContainerStarted","Data":"fe187294d6e0e60f07dac7bf93bfb5342a34f0b420cf634a4a1f5cba726266a8"}
Sep 05 00:10:48.289997 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:48.289884    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:48.304329 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:48.304286    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl" podStartSLOduration=1.304269495 podStartE2EDuration="1.304269495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 00:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 00:10:48.303645644 +0000 UTC m=+2060.550312722" watchObservedRunningTime="2026-09-05 00:10:48.304269495 +0000 UTC m=+2060.550936611"
Sep 05 00:10:48.578779 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:48.578750    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-vr466_9ad47c0d-7f00-4653-85fb-333ca37395d0/volume-data-source-validator/0.log"
Sep 05 00:10:49.696148 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:49.696120    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rv66q_f745f9b6-37b0-4710-a39f-282b4d388677/dns/0.log"
Sep 05 00:10:49.714616 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:49.714590    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rv66q_f745f9b6-37b0-4710-a39f-282b4d388677/kube-rbac-proxy/0.log"
Sep 05 00:10:49.786235 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:49.786205    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dpw8x_f507b59a-26e6-4c41-b0ea-5a428a8adba8/dns-node-resolver/0.log"
Sep 05 00:10:50.376544 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:50.376514    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pjgpk_0ffbbbd9-0ce0-4b3c-9a58-964ebeadf9a9/node-ca/0.log"
Sep 05 00:10:51.872005 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:51.871971    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d989d9b5-46mnh_c8007e7f-c503-42f0-a3b5-fdd289603da3/kube-auth-proxy/0.log"
Sep 05 00:10:51.923667 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:51.923620    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-ln8g4_d9907f4b-6c2c-4b46-b927-ff56ad289e45/istio-proxy/0.log"
Sep 05 00:10:51.940712 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:51.940683    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-ln8g4_d9907f4b-6c2c-4b46-b927-ff56ad289e45/istio-proxy/1.log"
Sep 05 00:10:52.022296 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:52.022263    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7nvhm_0b9fe449-ea11-4359-bf6e-e430169a274b/payload-processing/0.log"
Sep 05 00:10:52.049347 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:52.049312    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76b88fd7dd-5knhq_1b11f31d-54cf-49b7-a4b7-ae4984221a56/router/0.log"
Sep 05 00:10:52.864923 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:52.864883    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fmc2z_3d903250-92e3-48c2-b52f-e70ff30805a0/serve-healthcheck-canary/0.log"
Sep 05 00:10:53.363398 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:53.363344    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-vxcx6_b0cc507b-07c5-4ced-bb71-2cbc7b2c4f00/insights-operator/0.log"
Sep 05 00:10:53.384143 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:53.384112    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8xjqb_b41b8374-a266-47db-9ee5-13916d81025a/kube-rbac-proxy/0.log"
Sep 05 00:10:53.402598 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:53.402571    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8xjqb_b41b8374-a266-47db-9ee5-13916d81025a/exporter/0.log"
Sep 05 00:10:53.425187 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:53.425168    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8xjqb_b41b8374-a266-47db-9ee5-13916d81025a/extractor/0.log"
Sep 05 00:10:54.304839 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:54.304809    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-9j8gl"
Sep 05 00:10:55.550354 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:55.550320    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-57t66_f180203c-534c-4b4a-bc01-fb6885e26f22/manager/0.log"
Sep 05 00:10:55.793462 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:55.793432    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64c89c545-dw26z_762eb913-89ba-4212-80e8-1a7e591b2647/manager/0.log"
Sep 05 00:10:55.978029 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:55.977988    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b9496bc86-2xc4x_f1c5f4cf-5b1a-4aad-b528-592dcb82370f/manager/0.log"
Sep 05 00:10:55.996058 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:55.996030    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b9496bc86-2xc4x_f1c5f4cf-5b1a-4aad-b528-592dcb82370f/copy-manifests/0.log"
Sep 05 00:10:58.872403 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:58.872372    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d989d9b5-46mnh_c8007e7f-c503-42f0-a3b5-fdd289603da3/kube-auth-proxy/0.log"
Sep 05 00:10:58.917259 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:58.917228    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-ln8g4_d9907f4b-6c2c-4b46-b927-ff56ad289e45/istio-proxy/0.log"
Sep 05 00:10:58.933916 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:58.933873    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-ln8g4_d9907f4b-6c2c-4b46-b927-ff56ad289e45/istio-proxy/1.log"
Sep 05 00:10:58.992814 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:58.992780    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7nvhm_0b9fe449-ea11-4359-bf6e-e430169a274b/payload-processing/0.log"
Sep 05 00:10:59.009440 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:10:59.009414    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76b88fd7dd-5knhq_1b11f31d-54cf-49b7-a4b7-ae4984221a56/router/0.log"
Sep 05 00:11:01.007456 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:01.007416    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d989d9b5-46mnh_c8007e7f-c503-42f0-a3b5-fdd289603da3/kube-auth-proxy/0.log"
Sep 05 00:11:01.040140 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:01.040097    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-ln8g4_d9907f4b-6c2c-4b46-b927-ff56ad289e45/istio-proxy/0.log"
Sep 05 00:11:01.058630 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:01.058604    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-ln8g4_d9907f4b-6c2c-4b46-b927-ff56ad289e45/istio-proxy/1.log"
Sep 05 00:11:01.132767 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:01.132734    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7nvhm_0b9fe449-ea11-4359-bf6e-e430169a274b/payload-processing/0.log"
Sep 05 00:11:01.148201 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:01.148173    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76b88fd7dd-5knhq_1b11f31d-54cf-49b7-a4b7-ae4984221a56/router/0.log"
Sep 05 00:11:02.780962 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:02.780932    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-n22lt_506ee782-a460-4b93-afae-6a09bc31e2d3/migrator/0.log"
Sep 05 00:11:02.800147 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:02.800114    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-n22lt_506ee782-a460-4b93-afae-6a09bc31e2d3/graceful-termination/0.log"
Sep 05 00:11:03.156821 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:03.156736    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-bz687_2b6f2cdb-29a1-4245-83c3-1bf5eb182073/kube-storage-version-migrator-operator/1.log"
Sep 05 00:11:03.158247 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:03.158215    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-bz687_2b6f2cdb-29a1-4245-83c3-1bf5eb182073/kube-storage-version-migrator-operator/0.log"
Sep 05 00:11:04.122448 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:04.122422    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47qkp_133ae0cb-71c7-469e-bb5e-be538e8de7b3/kube-multus-additional-cni-plugins/0.log"
Sep 05 00:11:04.142426 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:04.142394    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47qkp_133ae0cb-71c7-469e-bb5e-be538e8de7b3/egress-router-binary-copy/0.log"
Sep 05 00:11:04.162183 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:04.162156    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47qkp_133ae0cb-71c7-469e-bb5e-be538e8de7b3/cni-plugins/0.log"
Sep 05 00:11:04.184186 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:04.184161    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47qkp_133ae0cb-71c7-469e-bb5e-be538e8de7b3/bond-cni-plugin/0.log"
Sep 05 00:11:04.203802 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:04.203777    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47qkp_133ae0cb-71c7-469e-bb5e-be538e8de7b3/routeoverride-cni/0.log"
Sep 05 00:11:04.223172 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:04.223153    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47qkp_133ae0cb-71c7-469e-bb5e-be538e8de7b3/whereabouts-cni-bincopy/0.log"
Sep 05 00:11:04.241858 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:04.241835    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47qkp_133ae0cb-71c7-469e-bb5e-be538e8de7b3/whereabouts-cni/0.log"
Sep 05 00:11:04.689468 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:04.689435    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-x2gpd_47ad65d3-8c19-41b5-b945-ed9944a35e94/kube-multus/0.log"
Sep 05 00:11:04.756360 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:04.756330    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m8mxs_38a67465-053c-4563-8c2c-8bff74b88a4b/network-metrics-daemon/0.log"
Sep 05 00:11:04.771314 ip-10-0-138-66 kubenswrapper[2562]: I0905 00:11:04.771286    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m8mxs_38a67465-053c-4563-8c2c-8bff74b88a4b/kube-rbac-proxy/0.log"