Sep 09 05:47:35.907500 ip-10-0-131-250 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 05:47:35.907511 ip-10-0-131-250 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 05:47:35.907519 ip-10-0-131-250 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 05:47:35.907665 ip-10-0-131-250 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 05:47:47.280408 ip-10-0-131-250 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 05:47:47.280425 ip-10-0-131-250 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f5b69411af8f4619b41391e2aec85a26 --
Sep 09 05:49:32.669808 ip-10-0-131-250 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 05:49:33.095965 ip-10-0-131-250 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 05:49:33.095965 ip-10-0-131-250 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 05:49:33.095965 ip-10-0-131-250 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 05:49:33.095965 ip-10-0-131-250 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 05:49:33.095965 ip-10-0-131-250 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 05:49:33.097979 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.097884    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 05:49:33.101777 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101761    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 05:49:33.101777 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101775    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 05:49:33.101777 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101779    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101784    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101787    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101790    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101793    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101796    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101799    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101802    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101804    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101812    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101815    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101819    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101823    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101826    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101829    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101833    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101836    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101838    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101841    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 05:49:33.101872 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101844    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101847    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101851    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101854    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101857    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101872    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101876    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101879    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101882    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101885    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101888    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101891    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101894    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101896    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101899    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101902    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101905    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101907    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101910    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 05:49:33.102357 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101913    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101915    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101918    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101920    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101923    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101926    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101928    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101931    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101934    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101936    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101939    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101941    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101944    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101947    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101950    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101953    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101956    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101958    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101961    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101964    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 05:49:33.102852 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101973    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101976    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101979    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101982    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101984    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101987    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101989    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101992    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101994    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.101998    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102001    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102003    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102006    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102008    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102011    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102013    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102016    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102018    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102021    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102023    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 05:49:33.103356 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102026    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102029    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102032    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102034    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102037    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102039    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102043    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102045    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102048    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102051    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102053    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102056    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102059    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102061    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102065    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102069    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102072    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102074    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102078    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102796    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 05:49:33.103846 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102803    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102806    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102810    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102812    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102815    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102818    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102822    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102826    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102829    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102832    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102836    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102838    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102841    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102843    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102846    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102849    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102851    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102854    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102857    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102860    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 05:49:33.104353 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102863    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102866    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102869    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102871    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102874    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102877    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102879    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102882    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102884    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102887    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102889    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102892    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102894    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102897    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102900    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102903    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102905    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102908    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102911    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102913    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 05:49:33.104892 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102916    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102918    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102921    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102924    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102926    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102929    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102931    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102934    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102936    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102939    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102942    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102944    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102947    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102950    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102955    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102959    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102962    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102965    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102968    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 05:49:33.105386 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102971    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102974    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102976    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102979    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102982    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102985    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102988    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102991    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102993    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102996    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.102999    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103001    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103004    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103006    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103009    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103012    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103015    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103017    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103020    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 05:49:33.105866 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103022    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103025    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103028    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103031    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103033    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103036    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103039    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103041    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103044    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103046    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103049    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103052    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103054    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103058    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103060    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103063    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103065    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103069    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103072    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103075    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 05:49:33.106349 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103432    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103437    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103441    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103443    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103446    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103449    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103451    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103454    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103456    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103459    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103462    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103465    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103468    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103470    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103473    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103476    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103479    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103481    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103484    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103486    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 05:49:33.106931 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103505    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103508    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103513    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103517    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103520    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103522    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103525    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103529    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103531    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103534    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103536    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103540    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103543    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103546    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103549    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103553    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103558    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103561    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103564    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 05:49:33.107422 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103566    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103569    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103571    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103574    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103576    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103580    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103582    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103586    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103588    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103591    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103594    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103599    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103602    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103604    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103607    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103609    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103612    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103614    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103617    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103619    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 05:49:33.107890 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103622    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103625    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103627    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103630    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103632    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103634    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103637    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103639    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103642    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103649    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103652    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103655    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103657    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103660    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103663    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103665    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103668    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103670    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103673    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103675    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 05:49:33.108380 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103678    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103681    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103683    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103685    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103688    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103690    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103693    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103695    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103698    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103700    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103703    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103706    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103708    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103711    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103714    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103716    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103718    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103721    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103724    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.103726    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103797    2564 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 05:49:33.108885 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103837    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103844    2564 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103849    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103854    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103857    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103862    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103866    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103870    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103873    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103876    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103880    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103883    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103886    2564 flags.go:64] FLAG: --cgroup-root=""
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103888    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103892    2564 flags.go:64] FLAG: --client-ca-file=""
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103894    2564 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103897    2564 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103901    2564 flags.go:64] FLAG: --cluster-domain=""
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103904    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103908    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103911    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103914    2564 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103918    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 05:49:33.109390 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103921    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103924    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103928    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103931    2564 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103934    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103937    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103940    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103943    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103947    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103950    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103953    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103956    2564 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103960    2564 flags.go:64] FLAG: --enable-server="true"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103963    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103967    2564 flags.go:64] FLAG: --event-burst="100"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103970    2564 flags.go:64] FLAG: --event-qps="50"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103973    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103976    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103979    2564 flags.go:64] FLAG: --eviction-hard=""
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103982    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103985    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103988    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103991    2564 flags.go:64] FLAG: --eviction-soft=""
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103994    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103997    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 05:49:33.109961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.103999    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104004    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104007    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104010    2564 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104012    2564 flags.go:64] FLAG: --feature-gates=""
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104016    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104019    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104022    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104025    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104028    2564 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104031    2564 flags.go:64] FLAG: --help="false"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104033    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104036    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104039    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104042    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104045    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104049    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104052    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104054    2564 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104061    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104064    2564 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104067    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104071    2564 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104073    2564 flags.go:64] FLAG: --kube-reserved=""
Sep 09 05:49:33.110632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104076    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104079    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104082    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104085    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104088    2564 flags.go:64] FLAG: --lock-file=""
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104090    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104093    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104096    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104101    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104106    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104109    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104112    2564 flags.go:64] FLAG: --logging-format="text"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104115    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104118    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104121    2564 flags.go:64] FLAG: --manifest-url=""
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104124    2564 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104128    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104131    2564 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104135    2564 flags.go:64] FLAG: --max-pods="110"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104138    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104141    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104144    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104147    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104150    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104153    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 05:49:33.111218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104156    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104163    2564 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104166    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104169    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104172    2564 flags.go:64] FLAG: --pod-cidr=""
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104175    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104180    2564 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104183    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104186    2564 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104189    2564 flags.go:64] FLAG: --port="10250"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104193    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104196    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f320b40deea76737"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104199    2564 flags.go:64] FLAG: --qos-reserved=""
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104202    2564 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104205    2564 flags.go:64] FLAG: --register-node="true"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104207    2564 flags.go:64] FLAG: --register-with-taints=""
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104212    2564 flags.go:64] FLAG: --registry-burst="10"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104215    2564 flags.go:64] FLAG: --registry-qps="5"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104218    2564 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104221    2564 flags.go:64] FLAG: --reserved-memory=""
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104224    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104227    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104230    2564 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104233    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104236    2564 flags.go:64] FLAG: --runonce="false"
Sep 09 05:49:33.112035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104239    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104242    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104245    2564 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104248    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104251    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104254    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104257    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104260    2564 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104263    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104266    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104269    2564 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104272    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104275    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104278    2564 flags.go:64] FLAG: --system-cgroups=""
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104281    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104286    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104289    2564 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104291    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104295    2564 flags.go:64] FLAG: --tls-min-version=""
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104298    2564 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104300    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104303    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104306    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104311    2564 flags.go:64] FLAG: --v="2"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104315    2564 flags.go:64] FLAG: --version="false"
Sep 09 05:49:33.112867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104319    2564 flags.go:64] FLAG: --vmodule=""
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104322    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.104325    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104447    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104451    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104454    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104457    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104460    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104462    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104465    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104468    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104470    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104473    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104476    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104479    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104482    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104484    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104487    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104507    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104511    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 05:49:33.113566 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104515    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104518    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104520    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104523    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104525    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104529    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104533    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104536    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104538    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104541    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104546    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104548    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104551    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104554    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104556    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104559    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104562    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104564    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104567    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104569    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 05:49:33.114401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104572    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104574    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104577    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104580    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104582    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104586    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104589    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104591    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104594    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104596    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104599    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104602    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104605    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104607    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104610    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104612    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104615    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104617    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104620    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 05:49:33.115138 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104623    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104625    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104629    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104635    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104638    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104640    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104643    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104646    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104648    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104651    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104654    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104656    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104659    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104661    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104664    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104667    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104669    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104672    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104675    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104678    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 05:49:33.115635 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104681    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104683    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104686    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104688    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104691    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104694    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104697    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104707    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104709    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104712    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104715    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104717    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104720    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104723    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104725    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104730    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104732    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104735    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104738    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104740    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 05:49:33.116447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104743    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 05:49:33.117171 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104745    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 05:49:33.117171 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.104748    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 05:49:33.117171 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.105483    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 05:49:33.117171 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.114791    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 05:49:33.117171 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.114817    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 05:49:33.117171 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.115681    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 05:49:33.117171 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.115695    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 05:49:33.117171 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115783    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 05:49:33.117171 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115791    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 05:49:33.117171 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115797    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 05:49:33.117171 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115802    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 05:49:33.117171 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115807    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 05:49:33.117171 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115812    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 05:49:33.117171 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115816    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 05:49:33.117171 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115821    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115826    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115831    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115837    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115842    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115846    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115851    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115855    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115859    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115863    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115867    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115871    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115876    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115880    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115884    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115889    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115893    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115897    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115901    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115905    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 05:49:33.117577 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115910    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115914    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115919    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115924    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115930    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115936    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115941    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115945    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115950    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115954    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115959    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115963    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115967    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115971    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115975    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.115979    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116028    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116073    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116079    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 05:49:33.118173 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116085    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116090    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116095    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116099    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116104    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116108    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116113    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116118    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116123    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116128    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116138    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116142    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.116147    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118078    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118231    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118236    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118239    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118244    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118248    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118251    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 05:49:33.118663 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118255    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118258    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118260    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118263    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118266    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118269    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118272    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118274    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118277    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118280    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118283    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118287    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118289    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118292    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118294    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118297    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118299    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118302    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118305    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118308    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 05:49:33.119188 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118310    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 05:49:33.119687 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118315    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 05:49:33.119687 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118320    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 05:49:33.119687 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118323    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 05:49:33.119687 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118326    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 05:49:33.119687 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118329    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 05:49:33.119687 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118332    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 05:49:33.119687 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118335    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 05:49:33.119687 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118337    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 05:49:33.119687 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118340    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 05:49:33.119687 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118343    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 05:49:33.119687 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118347    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 05:49:33.119687 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118351    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 05:49:33.119687 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.118357    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 05:49:33.119687 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118484    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118501    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118507    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118513    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118516    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118519    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118522    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118525    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118528    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118531    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118534    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118537    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118539    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118542    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118544    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118547    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118550    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118552    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118555    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118558    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 05:49:33.120034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118560    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118563    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118565    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118568    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118571    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118573    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118576    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118578    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118581    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118585    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118588    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118591    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118593    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118596    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118599    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118601    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118604    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118608    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118611    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 05:49:33.120528 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118614    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118617    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118620    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118623    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118626    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118629    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118632    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118634    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118637    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118639    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118642    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118644    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118648    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118650    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118653    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118655    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118658    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118661    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118663    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 05:49:33.120984 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118666    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118669    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118672    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118674    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118677    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118680    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118682    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118685    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118687    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118690    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118693    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118695    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118698    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118700    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118703    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118706    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118709    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118712    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118715    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118717    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 05:49:33.121449 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118720    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118723    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118725    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118728    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118730    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118733    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118736    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118738    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118740    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118743    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118746    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118748    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118751    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118753    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118758    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118760    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118763    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118766    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118768    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118771    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 05:49:33.122137 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:33.118773    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 05:49:33.122662 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.118778    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 05:49:33.122662 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.118889    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 05:49:33.122662 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.121170    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 05:49:33.122662 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.122307    2564 server.go:1019] "Starting client certificate rotation"
Sep 09 05:49:33.122662 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.122408    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 05:49:33.122662 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.122439    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 05:49:33.125985 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.125971    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 05:49:33.130528 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.130510    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 05:49:33.140263 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.140249    2564 log.go:25] "Validated CRI v1 runtime API"
Sep 09 05:49:33.143605 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.143589    2564 log.go:25] "Validated CRI v1 image API"
Sep 09 05:49:33.144486 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.144471    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 05:49:33.148706 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.148681    2564 fs.go:135] Filesystem UUIDs: map[15652c3b-d96d-400f-8070-1ffce8d8d388:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ca44f69c-68d7-4a2e-bcb4-0ab79d5bb0ba:/dev/nvme0n1p4]
Sep 09 05:49:33.148777 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.148705    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 05:49:33.154269 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.154244    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 05:49:33.155284 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.155174    2564 manager.go:216] Machine: {Timestamp:2026-09-09 05:49:33.15370915 +0000 UTC m=+0.370039024 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099246 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2251b8a5196100eafd73a750b0ac2f SystemUUID:ec2251b8-a519-6100-eafd-73a750b0ac2f BootID:f5b69411-af8f-4619-b413-91e2aec85a26 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:76:7f:53:bf:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:76:7f:53:bf:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:bb:96:86:f0:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 05:49:33.155284 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.155280    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 05:49:33.155412 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.155383    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 05:49:33.158969 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.158947    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 05:49:33.159642 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.158979    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-250.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 05:49:33.161023 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.161002    2564 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 05:49:33.161096 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.161028    2564 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 05:49:33.161096 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.161040    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 05:49:33.161096 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.161061    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 05:49:33.162996 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.162981    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 05:49:33.167598 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.167587    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 05:49:33.167709 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.167699    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 05:49:33.167747 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.167729    2564 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 05:49:33.167747 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.167739    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 05:49:33.167820 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.167753    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 05:49:33.167820 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.167762    2564 kubelet.go:398] "Adding apiserver pod source"
Sep 09 05:49:33.167820 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.167776    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 05:49:33.168189 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.168177    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 05:49:33.168222 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.168198    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 05:49:33.171294 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.171278    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 05:49:33.174131 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.174117    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 05:49:33.174195 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.174146    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 05:49:33.174294 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.174284    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 05:49:33.174341 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.174299    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 05:49:33.174341 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.174305    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 05:49:33.174341 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.174311    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 05:49:33.174341 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.174318    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 05:49:33.174341 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.174324    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 05:49:33.174977 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.174967    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 05:49:33.174977 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.174976    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 05:49:33.175078 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.174983    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 05:49:33.175078 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.174988    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 05:49:33.175078 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.175004    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 05:49:33.175078 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.175012    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 05:49:33.175920 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.175909    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 05:49:33.175920 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.175918    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 05:49:33.178692 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.178672    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-250.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 05:49:33.178790 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.178726    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 05:49:33.178867 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.178846    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 05:49:33.179867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.179855    2564 server.go:1268] "Started kubelet"
Sep 09 05:49:33.179923 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.179897    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 05:49:33.179973 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.179928    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 05:49:33.180021 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.179993    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 05:49:33.180770 ip-10-0-131-250 systemd[1]: Started Kubernetes Kubelet.
Sep 09 05:49:33.181531 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.181515    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 05:49:33.182807 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.182793    2564 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 05:49:33.184075 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.184053    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 05:49:33.186110 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.184956    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-250.ec2.internal.18d3910f995dfe10  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-250.ec2.internal,UID:ip-10-0-131-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-250.ec2.internal,},FirstTimestamp:2026-09-09 05:49:33.179829776 +0000 UTC m=+0.396159653,LastTimestamp:2026-09-09 05:49:33.179829776 +0000 UTC m=+0.396159653,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-250.ec2.internal,}"
Sep 09 05:49:33.188987 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.188965    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 05:49:33.189561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.189546    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 05:49:33.189707 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.189687    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 05:49:33.190354 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.190332    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 05:49:33.190354 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.190336    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 05:49:33.190354 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.190356    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 05:49:33.190525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.190416    2564 factory.go:55] Registering systemd factory
Sep 09 05:49:33.190525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.190428    2564 factory.go:223] Registration of the systemd container factory successfully
Sep 09 05:49:33.190525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.190441    2564 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 05:49:33.190525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.190448    2564 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 05:49:33.190736 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.190666    2564 factory.go:153] Registering CRI-O factory
Sep 09 05:49:33.190736 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.190682    2564 factory.go:223] Registration of the crio container factory successfully
Sep 09 05:49:33.190797 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.190754    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 05:49:33.190797 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.190773    2564 factory.go:103] Registering Raw factory
Sep 09 05:49:33.190872 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.190808    2564 manager.go:1196] Started watching for new ooms in manager
Sep 09 05:49:33.191380 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.191364    2564 manager.go:349] Starting recovery of all containers
Sep 09 05:49:33.198576 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.198553    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 05:49:33.198688 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.198569    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 05:49:33.202545 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.202360    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gdbbg"
Sep 09 05:49:33.204256 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.204238    2564 manager.go:354] Recovery completed
Sep 09 05:49:33.209807 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.209790    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gdbbg"
Sep 09 05:49:33.209901 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.209814    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 05:49:33.211748 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.211729    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 05:49:33.211832 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.211765    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 05:49:33.211832 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.211788    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientPID"
Sep 09 05:49:33.212437 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.212422    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 05:49:33.212437 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.212436    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 05:49:33.212559 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.212452    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 05:49:33.212559 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.212436    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 05:49:33.213216 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.213201    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 05:49:33.213289 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.213231    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 05:49:33.213289 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.213245    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientPID"
Sep 09 05:49:33.214552 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.214540    2564 policy_none.go:49] "None policy: Start"
Sep 09 05:49:33.214612 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.214555    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 05:49:33.214612 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.214565    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 05:49:33.216522 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.216511    2564 policy_none.go:47] "Start"
Sep 09 05:49:33.290336 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.290303    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 05:49:33.294310 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.294281    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 05:49:33.295532 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.295517    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 05:49:33.295595 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.295570    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 05:49:33.295640 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.295597    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 05:49:33.295732 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.295708    2564 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 09 05:49:33.298291 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.298274    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 05:49:33.368075 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.367993    2564 manager.go:337] "Starting Device Plugin manager"
Sep 09 05:49:33.368075 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.368030    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 05:49:33.368075 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.368040    2564 server.go:87] "Starting device plugin registration server"
Sep 09 05:49:33.368400 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.368386    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 05:49:33.368517 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.368398    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 05:49:33.368517 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.368510    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 05:49:33.368615 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.368592    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 05:49:33.368615 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.368602    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 05:49:33.369160 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.369142    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 05:49:33.369256 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.369188    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 05:49:33.396537 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.396513    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal"]
Sep 09 05:49:33.396639 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.396587    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 05:49:33.398320 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.398296    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 05:49:33.398407 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.398337    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 05:49:33.398407 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.398351    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientPID"
Sep 09 05:49:33.400722 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.400710    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 05:49:33.400836 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.400822    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.400874 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.400850    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 05:49:33.403662 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.403639    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 05:49:33.403662 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.403672    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 05:49:33.403789 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.403684    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientPID"
Sep 09 05:49:33.403944 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.403930    2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-250.ec2.internal\" not found" node="ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.404341 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.404329    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 05:49:33.404394 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.404351    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 05:49:33.404394 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.404361    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientPID"
Sep 09 05:49:33.405902 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.405883    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.405980 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.405905    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 05:49:33.406627 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.406611    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 05:49:33.406691 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.406638    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 05:49:33.406691 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.406648    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientPID"
Sep 09 05:49:33.433215 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.433193    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-250.ec2.internal\" not found" node="ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.437555 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.437539    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-250.ec2.internal\" not found" node="ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.469342 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.469326    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 05:49:33.470219 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.470204    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 05:49:33.470287 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.470230    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 05:49:33.470287 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.470240    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeHasSufficientPID"
Sep 09 05:49:33.470287 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.470267    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.479578 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.479561    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.479625 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.479582    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-250.ec2.internal\": node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 05:49:33.491522 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.491504    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b918f93a5380fe0ffc07a388ced93b00-config\") pod \"kube-apiserver-proxy-ip-10-0-131-250.ec2.internal\" (UID: \"b918f93a5380fe0ffc07a388ced93b00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.491591 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.491526    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1fb08136825eb2ae62fa73ecf8896a9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal\" (UID: \"1fb08136825eb2ae62fa73ecf8896a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.491591 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.491546    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fb08136825eb2ae62fa73ecf8896a9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal\" (UID: \"1fb08136825eb2ae62fa73ecf8896a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.502988 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.502972    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 05:49:33.592333 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.592293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b918f93a5380fe0ffc07a388ced93b00-config\") pod \"kube-apiserver-proxy-ip-10-0-131-250.ec2.internal\" (UID: \"b918f93a5380fe0ffc07a388ced93b00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.592418 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.592339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1fb08136825eb2ae62fa73ecf8896a9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal\" (UID: \"1fb08136825eb2ae62fa73ecf8896a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.592418 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.592366    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fb08136825eb2ae62fa73ecf8896a9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal\" (UID: \"1fb08136825eb2ae62fa73ecf8896a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.592418 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.592388    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b918f93a5380fe0ffc07a388ced93b00-config\") pod \"kube-apiserver-proxy-ip-10-0-131-250.ec2.internal\" (UID: \"b918f93a5380fe0ffc07a388ced93b00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.592418 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.592400    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fb08136825eb2ae62fa73ecf8896a9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal\" (UID: \"1fb08136825eb2ae62fa73ecf8896a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.592581 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.592406    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1fb08136825eb2ae62fa73ecf8896a9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal\" (UID: \"1fb08136825eb2ae62fa73ecf8896a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.603398 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.603377    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 05:49:33.704199 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.704131    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 05:49:33.736303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.736281    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.739794 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:33.739764    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal"
Sep 09 05:49:33.804684 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.804649    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 05:49:33.905098 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:33.905069    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 05:49:34.005618 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:34.005551    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 05:49:34.030470 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:34.030443    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 05:49:34.106477 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:34.106448    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 05:49:34.121601 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:34.121578    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 05:49:34.121758 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:34.121739    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 05:49:34.121815 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:34.121754    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 05:49:34.207062 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:34.207033    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 05:49:34.213185 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:34.213159    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 05:44:33 +0000 UTC" deadline="2028-03-18 19:01:33.681786395 +0000 UTC"
Sep 09 05:49:34.213238 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:34.213187    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13357h11m59.46860195s"
Sep 09 05:49:34.296256 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:34.296193    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 05:49:34.307133 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:34.307110    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 05:49:34.407608 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:34.407558    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 05:49:34.507985 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:34.507957    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-250.ec2.internal\" not found"
Sep 09 05:49:34.512505 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:34.512465    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb918f93a5380fe0ffc07a388ced93b00.slice/crio-38033d878abb9ccf879e8454794601152215d1f992e84cd878b0503f21fa4685 WatchSource:0}: Error finding container 38033d878abb9ccf879e8454794601152215d1f992e84cd878b0503f21fa4685: Status 404 returned error can't find the container with id 38033d878abb9ccf879e8454794601152215d1f992e84cd878b0503f21fa4685
Sep 09 05:49:34.516348 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:34.516329    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 05:49:34.599087 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:34.599026    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 05:49:34.690375 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:34.690346    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal"
Sep 09 05:49:34.698714 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:34.698695    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 05:49:34.699638 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:34.699625    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal"
Sep 09 05:49:34.711648 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:34.711630    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 05:49:35.169262 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.169231    2564 apiserver.go:52] "Watching apiserver"
Sep 09 05:49:35.176918 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.176735    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 05:49:35.177176 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.177150    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-dmx8r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal","openshift-multus/network-metrics-daemon-qgb26","openshift-network-diagnostics/network-check-target-zkc2s","kube-system/global-pull-secret-syncer-h8x9t","kube-system/konnectivity-agent-tz58z","openshift-multus/multus-additional-cni-plugins-wrz2h","openshift-multus/multus-xlfg6","openshift-network-operator/iptables-alerter-rjd2g","openshift-ovn-kubernetes/ovnkube-node-xm7sb","kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v","openshift-cluster-node-tuning-operator/tuned-f7clt","openshift-dns/node-resolver-h8v7x"]
Sep 09 05:49:35.180696 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.179792    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.182607 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.182583    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.184297 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.183916    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 05:49:35.184297 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.184161    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h2966\""
Sep 09 05:49:35.184447 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.184436    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 05:49:35.184718 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.184696    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 05:49:35.184935 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.184913    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 05:49:35.185044 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.184978    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:35.185102 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.184913    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 05:49:35.185102 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.185040    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:49:35.185336 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.185320    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 05:49:35.185407 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.185368    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 05:49:35.185688 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.185671    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hckmc\""
Sep 09 05:49:35.186476 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.186457    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:35.186579 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.186537    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:49:35.188915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.188882    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:35.189013 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.188945    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:49:35.189202 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.189094    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 05:49:35.191005 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.190990    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tz58z"
Sep 09 05:49:35.193202 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.193188    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rjd2g"
Sep 09 05:49:35.194308 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.194206    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 05:49:35.194308 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.194280    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 05:49:35.194464 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.194408    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8nhk5\""
Sep 09 05:49:35.197353 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.195549    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 05:49:35.197353 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.195829    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 05:49:35.197353 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.196057    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8l7lw\""
Sep 09 05:49:35.197353 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.196217    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 05:49:35.197353 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.196766    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dmx8r"
Sep 09 05:49:35.198731 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.198712    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 05:49:35.198946 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.198926    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzc46\" (UniqueName: \"kubernetes.io/projected/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-kube-api-access-qzc46\") pod \"network-metrics-daemon-qgb26\" (UID: \"910ab0f9-8007-4ec5-b5b9-bf0e32c85593\") " pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:35.199021 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.198961    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-dbus\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:35.199021 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.198986    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p825l\" (UniqueName: \"kubernetes.io/projected/9d280b22-a809-4323-8a18-2049d0394312-kube-api-access-p825l\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.199116 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-sysconfig\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.199116 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199083    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-kubelet-config\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:35.199210 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199113    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d280b22-a809-4323-8a18-2049d0394312-cnibin\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.199210 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199136    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d280b22-a809-4323-8a18-2049d0394312-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.199210 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199196    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d280b22-a809-4323-8a18-2049d0394312-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.199210 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199204    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h8v7x"
Sep 09 05:49:35.199383 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199220    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-sysctl-d\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.199383 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199236    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-tuned\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.199383 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199254    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkrqj\" (UniqueName: \"kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj\") pod \"network-check-target-zkc2s\" (UID: \"4fb7e960-b0a8-4ba1-88b7-616e4910732a\") " pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:35.199383 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199270    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c67f7d9-3ba8-43ad-bc02-a063ab1f8c91-agent-certs\") pod \"konnectivity-agent-tz58z\" (UID: \"3c67f7d9-3ba8-43ad-bc02-a063ab1f8c91\") " pod="kube-system/konnectivity-agent-tz58z"
Sep 09 05:49:35.199383 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199287    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-modprobe-d\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.199383 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199310    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-run\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.199383 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199350    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-host\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.199383 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199372    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d87a3026-ed95-4af8-b013-fc6640e5ce48-tmp\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.199684 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199407    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:35.199684 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199439    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d280b22-a809-4323-8a18-2049d0394312-cni-binary-copy\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.199684 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199462    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-var-lib-kubelet\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.199684 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199527    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c67f7d9-3ba8-43ad-bc02-a063ab1f8c91-konnectivity-ca\") pod \"konnectivity-agent-tz58z\" (UID: \"3c67f7d9-3ba8-43ad-bc02-a063ab1f8c91\") " pod="kube-system/konnectivity-agent-tz58z"
Sep 09 05:49:35.199684 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199570    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-sysctl-conf\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.199684 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199611    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-systemd\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.199684 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199656    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d280b22-a809-4323-8a18-2049d0394312-system-cni-dir\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.199684 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199682    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-kubernetes\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.199957 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199718    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gzrt\" (UniqueName: \"kubernetes.io/projected/d87a3026-ed95-4af8-b013-fc6640e5ce48-kube-api-access-5gzrt\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.199957 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs\") pod \"network-metrics-daemon-qgb26\" (UID: \"910ab0f9-8007-4ec5-b5b9-bf0e32c85593\") " pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:35.199957 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199783    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-sys\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.199957 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199819    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-lib-modules\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.203923 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.203869    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 05:49:35.204066 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.204046    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 05:49:35.204124 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.199852    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d280b22-a809-4323-8a18-2049d0394312-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.204550 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.204532    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9j88h\""
Sep 09 05:49:35.206002 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.205479    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 05:49:35.206002 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.205729    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nm762\""
Sep 09 05:49:35.206002 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.205764    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 05:49:35.206230 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.206206    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.208064 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.208015    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 05:49:35.210136 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.210115    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.210433 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.210414    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 05:49:35.210577 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.210464    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 05:49:35.210845 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.210825    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 05:49:35.210845 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.210841    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sdkmc\""
Sep 09 05:49:35.210980 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.210955    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 05:49:35.211051 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.210838    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 05:49:35.211612 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.211452    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 05:49:35.214098 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.212771    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.214098 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.213854    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2kj79\""
Sep 09 05:49:35.214265 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.214112    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 05:49:35.215744 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.214789    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 05:49:35.215744 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.215153    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wbfsk\""
Sep 09 05:49:35.215744 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.215344    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 05:49:35.215940 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.215826    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 05:49:35.255577 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.255554    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m6zz5"
Sep 09 05:49:35.268116 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.268094    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m6zz5"
Sep 09 05:49:35.291782 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.291724    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 05:49:35.301754 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.301662    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal" event={"ID":"b918f93a5380fe0ffc07a388ced93b00","Type":"ContainerStarted","Data":"38033d878abb9ccf879e8454794601152215d1f992e84cd878b0503f21fa4685"}
Sep 09 05:49:35.303803 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.303775    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal" event={"ID":"1fb08136825eb2ae62fa73ecf8896a9e","Type":"ContainerStarted","Data":"80ef52b77a15bbd4237ef561ed7e4d591f14198559693e184f55251ee7ad245e"}
Sep 09 05:49:35.304967 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.304945    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-kubelet-config\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:35.305062 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.304981    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d280b22-a809-4323-8a18-2049d0394312-cnibin\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.305062 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305009    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d280b22-a809-4323-8a18-2049d0394312-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.305062 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305032    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-sysctl-d\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.305062 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305054    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-tuned\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.305260 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305062    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-kubelet-config\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:35.305260 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305080    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40030ec2-6b02-4d13-b794-aa551a09290e-hosts-file\") pod \"node-resolver-h8v7x\" (UID: \"40030ec2-6b02-4d13-b794-aa551a09290e\") " pod="openshift-dns/node-resolver-h8v7x"
Sep 09 05:49:35.305260 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305098    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d280b22-a809-4323-8a18-2049d0394312-cnibin\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.305260 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-slash\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.305260 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgx29\" (UniqueName: \"kubernetes.io/projected/46f7dac2-803f-4c17-a41f-af2157f1670e-kube-api-access-tgx29\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.305260 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305187    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-sysctl-d\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.305260 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305207    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c67f7d9-3ba8-43ad-bc02-a063ab1f8c91-agent-certs\") pod \"konnectivity-agent-tz58z\" (UID: \"3c67f7d9-3ba8-43ad-bc02-a063ab1f8c91\") " pod="kube-system/konnectivity-agent-tz58z"
Sep 09 05:49:35.305260 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305229    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-run\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.305260 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305244    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d87a3026-ed95-4af8-b013-fc6640e5ce48-tmp\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.305628 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305306    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-run\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.305628 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305410    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-system-cni-dir\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.305628 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305447    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-run-netns\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.305628 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305469    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.305628 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305511    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-device-dir\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.305628 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305537    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:35.305628 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305563    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d280b22-a809-4323-8a18-2049d0394312-cni-binary-copy\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.305628 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305590    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-var-lib-kubelet\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.305628 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305614    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-run-openvswitch\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.306051 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305641    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-etc-kubernetes\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.306051 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.305661    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:35.306051 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-systemd\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.306051 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.305740    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret podName:9e1d1039-4eb0-4691-bd7f-21dfdd3870ca nodeName:}" failed. No retries permitted until 2026-09-09 05:49:35.805716531 +0000 UTC m=+3.022046410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret") pod "global-pull-secret-syncer-h8x9t" (UID: "9e1d1039-4eb0-4691-bd7f-21dfdd3870ca") : object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:35.306051 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305772    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8199db41-6f04-4d2e-a6ac-2d51fff9ab42-iptables-alerter-script\") pod \"iptables-alerter-rjd2g\" (UID: \"8199db41-6f04-4d2e-a6ac-2d51fff9ab42\") " pod="openshift-network-operator/iptables-alerter-rjd2g"
Sep 09 05:49:35.306051 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305787    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-var-lib-kubelet\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.306051 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305835    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d280b22-a809-4323-8a18-2049d0394312-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.306051 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305816    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 05:49:35.306051 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305884    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4234e30-8f0b-47d1-a67d-36deddc3a89c-serviceca\") pod \"node-ca-dmx8r\" (UID: \"f4234e30-8f0b-47d1-a67d-36deddc3a89c\") " pod="openshift-image-registry/node-ca-dmx8r"
Sep 09 05:49:35.306051 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.305975    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-sys-fs\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306046    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-kubernetes\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306081    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-var-lib-openvswitch\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306104    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-systemd\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306117    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-run-k8s-cni-cncf-io\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306140    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-var-lib-cni-multus\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306161    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xtzz\" (UniqueName: \"kubernetes.io/projected/f4234e30-8f0b-47d1-a67d-36deddc3a89c-kube-api-access-7xtzz\") pod \"node-ca-dmx8r\" (UID: \"f4234e30-8f0b-47d1-a67d-36deddc3a89c\") " pod="openshift-image-registry/node-ca-dmx8r"
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306184    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs\") pod \"network-metrics-daemon-qgb26\" (UID: \"910ab0f9-8007-4ec5-b5b9-bf0e32c85593\") " pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306182    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-kubernetes\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306219    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-sys\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306261    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-run-ovn\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306283    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-multus-cni-dir\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306301    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-kubelet\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.306306    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306328    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-cni-bin\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.306355    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs podName:910ab0f9-8007-4ec5-b5b9-bf0e32c85593 nodeName:}" failed. No retries permitted until 2026-09-09 05:49:35.80633776 +0000 UTC m=+3.022667637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs") pod "network-metrics-daemon-qgb26" (UID: "910ab0f9-8007-4ec5-b5b9-bf0e32c85593") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306385    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-sys\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.306561 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306420    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46f7dac2-803f-4c17-a41f-af2157f1670e-ovnkube-config\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.307272 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306456    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46f7dac2-803f-4c17-a41f-af2157f1670e-env-overrides\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.307272 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306486    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46f7dac2-803f-4c17-a41f-af2157f1670e-ovn-node-metrics-cert\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.307272 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306526    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-multus-socket-dir-parent\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.307272 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-var-lib-cni-bin\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.307272 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306579    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-hostroot\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.307272 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306632    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzc46\" (UniqueName: \"kubernetes.io/projected/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-kube-api-access-qzc46\") pod \"network-metrics-daemon-qgb26\" (UID: \"910ab0f9-8007-4ec5-b5b9-bf0e32c85593\") " pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:35.307272 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p825l\" (UniqueName: \"kubernetes.io/projected/9d280b22-a809-4323-8a18-2049d0394312-kube-api-access-p825l\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.307272 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306884    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-run-systemd\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.307272 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306910    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-etc-openvswitch\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.307272 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306943    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-node-log\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.307272 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306955    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d280b22-a809-4323-8a18-2049d0394312-cni-binary-copy\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.307272 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.306991    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d280b22-a809-4323-8a18-2049d0394312-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.307272 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307242    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307278    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-cnibin\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrqj\" (UniqueName: \"kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj\") pod \"network-check-target-zkc2s\" (UID: \"4fb7e960-b0a8-4ba1-88b7-616e4910732a\") " pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307345    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-modprobe-d\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307446    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-host\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307501    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-run-netns\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307562    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46f7dac2-803f-4c17-a41f-af2157f1670e-ovnkube-script-lib\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307561    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d280b22-a809-4323-8a18-2049d0394312-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307598    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2rrf\" (UniqueName: \"kubernetes.io/projected/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-kube-api-access-j2rrf\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307656    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-modprobe-d\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307672    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-host\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307679    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8199db41-6f04-4d2e-a6ac-2d51fff9ab42-host-slash\") pod \"iptables-alerter-rjd2g\" (UID: \"8199db41-6f04-4d2e-a6ac-2d51fff9ab42\") " pod="openshift-network-operator/iptables-alerter-rjd2g"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-log-socket\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307748    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4234e30-8f0b-47d1-a67d-36deddc3a89c-host\") pod \"node-ca-dmx8r\" (UID: \"f4234e30-8f0b-47d1-a67d-36deddc3a89c\") " pod="openshift-image-registry/node-ca-dmx8r"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307775    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c67f7d9-3ba8-43ad-bc02-a063ab1f8c91-konnectivity-ca\") pod \"konnectivity-agent-tz58z\" (UID: \"3c67f7d9-3ba8-43ad-bc02-a063ab1f8c91\") " pod="kube-system/konnectivity-agent-tz58z"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-sysctl-conf\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.307943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307921    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr4lw\" (UniqueName: \"kubernetes.io/projected/8199db41-6f04-4d2e-a6ac-2d51fff9ab42-kube-api-access-rr4lw\") pod \"iptables-alerter-rjd2g\" (UID: \"8199db41-6f04-4d2e-a6ac-2d51fff9ab42\") " pod="openshift-network-operator/iptables-alerter-rjd2g"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.307970    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d280b22-a809-4323-8a18-2049d0394312-system-cni-dir\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308004    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-sysctl-conf\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308007    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gzrt\" (UniqueName: \"kubernetes.io/projected/d87a3026-ed95-4af8-b013-fc6640e5ce48-kube-api-access-5gzrt\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308062    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-systemd-units\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308096    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308222    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-cni-binary-copy\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308281    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-var-lib-kubelet\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308307    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-run-multus-certs\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308336    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d280b22-a809-4323-8a18-2049d0394312-system-cni-dir\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308280    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c67f7d9-3ba8-43ad-bc02-a063ab1f8c91-konnectivity-ca\") pod \"konnectivity-agent-tz58z\" (UID: \"3c67f7d9-3ba8-43ad-bc02-a063ab1f8c91\") " pod="kube-system/konnectivity-agent-tz58z"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308340    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-registration-dir\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308399    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-lib-modules\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308430    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40030ec2-6b02-4d13-b794-aa551a09290e-tmp-dir\") pod \"node-resolver-h8v7x\" (UID: \"40030ec2-6b02-4d13-b794-aa551a09290e\") " pod="openshift-dns/node-resolver-h8v7x"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308460    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-cni-netd\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308485    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308527    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-lib-modules\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.309446 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308536    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r2fg\" (UniqueName: \"kubernetes.io/projected/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-kube-api-access-9r2fg\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.309446 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308563    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d280b22-a809-4323-8a18-2049d0394312-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.309446 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308586    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-dbus\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:35.309446 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308608    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-sysconfig\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.309446 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308631    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sp6n\" (UniqueName: \"kubernetes.io/projected/40030ec2-6b02-4d13-b794-aa551a09290e-kube-api-access-4sp6n\") pod \"node-resolver-h8v7x\" (UID: \"40030ec2-6b02-4d13-b794-aa551a09290e\") " pod="openshift-dns/node-resolver-h8v7x"
Sep 09 05:49:35.309446 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308653    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-multus-conf-dir\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.309446 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308677    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-dbus\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:35.309446 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308689    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-multus-daemon-config\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.309446 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308746    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-socket-dir\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.309446 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.308860    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-sysconfig\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.309446 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.309055    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d280b22-a809-4323-8a18-2049d0394312-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.310257 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.310236    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d87a3026-ed95-4af8-b013-fc6640e5ce48-tmp\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.310554 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.310534    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c67f7d9-3ba8-43ad-bc02-a063ab1f8c91-agent-certs\") pod \"konnectivity-agent-tz58z\" (UID: \"3c67f7d9-3ba8-43ad-bc02-a063ab1f8c91\") " pod="kube-system/konnectivity-agent-tz58z"
Sep 09 05:49:35.312066 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.312046    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d87a3026-ed95-4af8-b013-fc6640e5ce48-etc-tuned\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.322767 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.322742    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gzrt\" (UniqueName: \"kubernetes.io/projected/d87a3026-ed95-4af8-b013-fc6640e5ce48-kube-api-access-5gzrt\") pod \"tuned-f7clt\" (UID: \"d87a3026-ed95-4af8-b013-fc6640e5ce48\") " pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.323561 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.323542    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 05:49:35.323646 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.323565    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 05:49:35.323646 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.323579    2564 projected.go:196] Error preparing data for projected volume kube-api-access-zkrqj for pod openshift-network-diagnostics/network-check-target-zkc2s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:35.323748 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.323649    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj podName:4fb7e960-b0a8-4ba1-88b7-616e4910732a nodeName:}" failed. No retries permitted until 2026-09-09 05:49:35.823623196 +0000 UTC m=+3.039953081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zkrqj" (UniqueName: "kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj") pod "network-check-target-zkc2s" (UID: "4fb7e960-b0a8-4ba1-88b7-616e4910732a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:35.327124 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.327048    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzc46\" (UniqueName: \"kubernetes.io/projected/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-kube-api-access-qzc46\") pod \"network-metrics-daemon-qgb26\" (UID: \"910ab0f9-8007-4ec5-b5b9-bf0e32c85593\") " pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:35.327215 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.327116    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p825l\" (UniqueName: \"kubernetes.io/projected/9d280b22-a809-4323-8a18-2049d0394312-kube-api-access-p825l\") pod \"multus-additional-cni-plugins-wrz2h\" (UID: \"9d280b22-a809-4323-8a18-2049d0394312\") " pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.409480 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409445    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-var-lib-openvswitch\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.409646 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-run-k8s-cni-cncf-io\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.409646 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409554    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-run-k8s-cni-cncf-io\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.409646 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409596    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-var-lib-openvswitch\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.409646 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409551    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-var-lib-cni-multus\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.409646 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409644    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xtzz\" (UniqueName: \"kubernetes.io/projected/f4234e30-8f0b-47d1-a67d-36deddc3a89c-kube-api-access-7xtzz\") pod \"node-ca-dmx8r\" (UID: \"f4234e30-8f0b-47d1-a67d-36deddc3a89c\") " pod="openshift-image-registry/node-ca-dmx8r"
Sep 09 05:49:35.409921 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409675    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-var-lib-cni-multus\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.409921 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409684    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-run-ovn\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.409921 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409709    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-multus-cni-dir\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.409921 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409740    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-kubelet\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.409921 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409765    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-cni-bin\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.409921 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409773    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-multus-cni-dir\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.409921 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409808    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-cni-bin\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.409921 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409812    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-kubelet\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.409921 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409819    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-run-ovn\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.409921 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409858    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46f7dac2-803f-4c17-a41f-af2157f1670e-ovnkube-config\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.409921 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409885    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46f7dac2-803f-4c17-a41f-af2157f1670e-env-overrides\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.409921 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409911    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46f7dac2-803f-4c17-a41f-af2157f1670e-ovn-node-metrics-cert\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409934    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-multus-socket-dir-parent\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409958    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-var-lib-cni-bin\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.409980    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-hostroot\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410006    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-run-systemd\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-etc-openvswitch\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410069    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-var-lib-cni-bin\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410079    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-etc-openvswitch\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410103    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-node-log\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410119    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-hostroot\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410156    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-cnibin\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410162    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-run-systemd\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410210    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-cnibin\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410250    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-run-netns\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410274    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46f7dac2-803f-4c17-a41f-af2157f1670e-ovnkube-script-lib\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410300    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2rrf\" (UniqueName: \"kubernetes.io/projected/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-kube-api-access-j2rrf\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410323    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8199db41-6f04-4d2e-a6ac-2d51fff9ab42-host-slash\") pod \"iptables-alerter-rjd2g\" (UID: \"8199db41-6f04-4d2e-a6ac-2d51fff9ab42\") " pod="openshift-network-operator/iptables-alerter-rjd2g"
Sep 09 05:49:35.410525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410350    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-log-socket\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410371    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4234e30-8f0b-47d1-a67d-36deddc3a89c-host\") pod \"node-ca-dmx8r\" (UID: \"f4234e30-8f0b-47d1-a67d-36deddc3a89c\") " pod="openshift-image-registry/node-ca-dmx8r"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410397    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr4lw\" (UniqueName: \"kubernetes.io/projected/8199db41-6f04-4d2e-a6ac-2d51fff9ab42-kube-api-access-rr4lw\") pod \"iptables-alerter-rjd2g\" (UID: \"8199db41-6f04-4d2e-a6ac-2d51fff9ab42\") " pod="openshift-network-operator/iptables-alerter-rjd2g"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410423    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-systemd-units\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410455    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410480    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-cni-binary-copy\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410522    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-var-lib-kubelet\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410546    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-run-multus-certs\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410569    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-registration-dir\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410593    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40030ec2-6b02-4d13-b794-aa551a09290e-tmp-dir\") pod \"node-resolver-h8v7x\" (UID: \"40030ec2-6b02-4d13-b794-aa551a09290e\") " pod="openshift-dns/node-resolver-h8v7x"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410615    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-cni-netd\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410640    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r2fg\" (UniqueName: \"kubernetes.io/projected/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-kube-api-access-9r2fg\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410692    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sp6n\" (UniqueName: \"kubernetes.io/projected/40030ec2-6b02-4d13-b794-aa551a09290e-kube-api-access-4sp6n\") pod \"node-resolver-h8v7x\" (UID: \"40030ec2-6b02-4d13-b794-aa551a09290e\") " pod="openshift-dns/node-resolver-h8v7x"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410716    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-multus-conf-dir\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410738    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-multus-daemon-config\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410764    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-socket-dir\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.411416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410795    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40030ec2-6b02-4d13-b794-aa551a09290e-hosts-file\") pod \"node-resolver-h8v7x\" (UID: \"40030ec2-6b02-4d13-b794-aa551a09290e\") " pod="openshift-dns/node-resolver-h8v7x"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410820    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-slash\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410845    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgx29\" (UniqueName: \"kubernetes.io/projected/46f7dac2-803f-4c17-a41f-af2157f1670e-kube-api-access-tgx29\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410876    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-system-cni-dir\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410900    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-run-netns\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410925    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410949    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-device-dir\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.410950    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46f7dac2-803f-4c17-a41f-af2157f1670e-env-overrides\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411001    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46f7dac2-803f-4c17-a41f-af2157f1670e-ovnkube-config\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411012    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-run-openvswitch\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411051    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-etc-kubernetes\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411082    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8199db41-6f04-4d2e-a6ac-2d51fff9ab42-iptables-alerter-script\") pod \"iptables-alerter-rjd2g\" (UID: \"8199db41-6f04-4d2e-a6ac-2d51fff9ab42\") " pod="openshift-network-operator/iptables-alerter-rjd2g"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411086    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-cni-netd\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411108    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4234e30-8f0b-47d1-a67d-36deddc3a89c-serviceca\") pod \"node-ca-dmx8r\" (UID: \"f4234e30-8f0b-47d1-a67d-36deddc3a89c\") " pod="openshift-image-registry/node-ca-dmx8r"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411133    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-sys-fs\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411180    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40030ec2-6b02-4d13-b794-aa551a09290e-tmp-dir\") pod \"node-resolver-h8v7x\" (UID: \"40030ec2-6b02-4d13-b794-aa551a09290e\") " pod="openshift-dns/node-resolver-h8v7x"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411201    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.412303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411224    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-sys-fs\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411232    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-run-netns\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411053    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-run-openvswitch\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411292    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-log-socket\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411335    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4234e30-8f0b-47d1-a67d-36deddc3a89c-host\") pod \"node-ca-dmx8r\" (UID: \"f4234e30-8f0b-47d1-a67d-36deddc3a89c\") " pod="openshift-image-registry/node-ca-dmx8r"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411578    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-systemd-units\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411624    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411705    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-multus-conf-dir\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411810    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8199db41-6f04-4d2e-a6ac-2d51fff9ab42-iptables-alerter-script\") pod \"iptables-alerter-rjd2g\" (UID: \"8199db41-6f04-4d2e-a6ac-2d51fff9ab42\") " pod="openshift-network-operator/iptables-alerter-rjd2g"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411858    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-etc-kubernetes\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411855    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-run-multus-certs\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411892    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8199db41-6f04-4d2e-a6ac-2d51fff9ab42-host-slash\") pod \"iptables-alerter-rjd2g\" (UID: \"8199db41-6f04-4d2e-a6ac-2d51fff9ab42\") " pod="openshift-network-operator/iptables-alerter-rjd2g"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.411914    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-var-lib-kubelet\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.412173    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-cni-binary-copy\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.412203    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-socket-dir\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.412242    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-registration-dir\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.412276    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40030ec2-6b02-4d13-b794-aa551a09290e-hosts-file\") pod \"node-resolver-h8v7x\" (UID: \"40030ec2-6b02-4d13-b794-aa551a09290e\") " pod="openshift-dns/node-resolver-h8v7x"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.412284    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-node-log\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.413030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.412300    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4234e30-8f0b-47d1-a67d-36deddc3a89c-serviceca\") pod \"node-ca-dmx8r\" (UID: \"f4234e30-8f0b-47d1-a67d-36deddc3a89c\") " pod="openshift-image-registry/node-ca-dmx8r"
Sep 09 05:49:35.413951 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.412317    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46f7dac2-803f-4c17-a41f-af2157f1670e-ovnkube-script-lib\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.413951 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.412330    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-multus-socket-dir-parent\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.413951 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.412322    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-slash\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.413951 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.412344    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7dac2-803f-4c17-a41f-af2157f1670e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.413951 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.412383    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-system-cni-dir\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.413951 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.412389    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.413951 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.412393    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-device-dir\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.413951 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.412442    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-host-run-netns\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.413951 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.412802    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-multus-daemon-config\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.413951 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.413509    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46f7dac2-803f-4c17-a41f-af2157f1670e-ovn-node-metrics-cert\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.418188 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.418161    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xtzz\" (UniqueName: \"kubernetes.io/projected/f4234e30-8f0b-47d1-a67d-36deddc3a89c-kube-api-access-7xtzz\") pod \"node-ca-dmx8r\" (UID: \"f4234e30-8f0b-47d1-a67d-36deddc3a89c\") " pod="openshift-image-registry/node-ca-dmx8r"
Sep 09 05:49:35.420449 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.420392    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr4lw\" (UniqueName: \"kubernetes.io/projected/8199db41-6f04-4d2e-a6ac-2d51fff9ab42-kube-api-access-rr4lw\") pod \"iptables-alerter-rjd2g\" (UID: \"8199db41-6f04-4d2e-a6ac-2d51fff9ab42\") " pod="openshift-network-operator/iptables-alerter-rjd2g"
Sep 09 05:49:35.420783 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.420761    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r2fg\" (UniqueName: \"kubernetes.io/projected/44fcf3f6-42e5-4af7-9b85-cb0f1935d39c-kube-api-access-9r2fg\") pod \"aws-ebs-csi-driver-node-hdb9v\" (UID: \"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.421248 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.421226    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2rrf\" (UniqueName: \"kubernetes.io/projected/0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c-kube-api-access-j2rrf\") pod \"multus-xlfg6\" (UID: \"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c\") " pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.421710 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.421683    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgx29\" (UniqueName: \"kubernetes.io/projected/46f7dac2-803f-4c17-a41f-af2157f1670e-kube-api-access-tgx29\") pod \"ovnkube-node-xm7sb\" (UID: \"46f7dac2-803f-4c17-a41f-af2157f1670e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.422668 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.422622    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sp6n\" (UniqueName: \"kubernetes.io/projected/40030ec2-6b02-4d13-b794-aa551a09290e-kube-api-access-4sp6n\") pod \"node-resolver-h8v7x\" (UID: \"40030ec2-6b02-4d13-b794-aa551a09290e\") " pod="openshift-dns/node-resolver-h8v7x"
Sep 09 05:49:35.495162 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.495129    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wrz2h"
Sep 09 05:49:35.506583 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.506554    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f7clt"
Sep 09 05:49:35.509724 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:35.509694    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d280b22_a809_4323_8a18_2049d0394312.slice/crio-2a80cd46a5c621075e1dc1f97fea8e545840c3da147698026a9491b44a8a653d WatchSource:0}: Error finding container 2a80cd46a5c621075e1dc1f97fea8e545840c3da147698026a9491b44a8a653d: Status 404 returned error can't find the container with id 2a80cd46a5c621075e1dc1f97fea8e545840c3da147698026a9491b44a8a653d
Sep 09 05:49:35.516345 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:35.516317    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87a3026_ed95_4af8_b013_fc6640e5ce48.slice/crio-eedad4ed7c2ee5bc8105e47e70d2ac3939484d3bb2ae853860f08aa5b7fe6dd3 WatchSource:0}: Error finding container eedad4ed7c2ee5bc8105e47e70d2ac3939484d3bb2ae853860f08aa5b7fe6dd3: Status 404 returned error can't find the container with id eedad4ed7c2ee5bc8105e47e70d2ac3939484d3bb2ae853860f08aa5b7fe6dd3
Sep 09 05:49:35.517506 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.517470    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tz58z"
Sep 09 05:49:35.527324 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.527123    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rjd2g"
Sep 09 05:49:35.527514 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:35.527472    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c67f7d9_3ba8_43ad_bc02_a063ab1f8c91.slice/crio-61ec9d33b82d8eb40a20938241b7656f338c0fa044c6d41a678c26577179aa69 WatchSource:0}: Error finding container 61ec9d33b82d8eb40a20938241b7656f338c0fa044c6d41a678c26577179aa69: Status 404 returned error can't find the container with id 61ec9d33b82d8eb40a20938241b7656f338c0fa044c6d41a678c26577179aa69
Sep 09 05:49:35.533400 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.533381    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dmx8r"
Sep 09 05:49:35.535889 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:35.535866    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8199db41_6f04_4d2e_a6ac_2d51fff9ab42.slice/crio-6cba0664fc2cb03b10d8b29c6c458f642c49a3c1e084092ceed41f5cc3b773f2 WatchSource:0}: Error finding container 6cba0664fc2cb03b10d8b29c6c458f642c49a3c1e084092ceed41f5cc3b773f2: Status 404 returned error can't find the container with id 6cba0664fc2cb03b10d8b29c6c458f642c49a3c1e084092ceed41f5cc3b773f2
Sep 09 05:49:35.542036 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:35.542017    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4234e30_8f0b_47d1_a67d_36deddc3a89c.slice/crio-d8260f8e71da9c03d935a94134eabd85be12f52a50babb3b6d4da3966b27e243 WatchSource:0}: Error finding container d8260f8e71da9c03d935a94134eabd85be12f52a50babb3b6d4da3966b27e243: Status 404 returned error can't find the container with id d8260f8e71da9c03d935a94134eabd85be12f52a50babb3b6d4da3966b27e243
Sep 09 05:49:35.542928 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.542906    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h8v7x"
Sep 09 05:49:35.549729 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.549710    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:35.553593 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:35.553565    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40030ec2_6b02_4d13_b794_aa551a09290e.slice/crio-f7a6e0d1b195e6d6ce9d5ee61bd023aeb61197187807442501b3e24b68b87ff6 WatchSource:0}: Error finding container f7a6e0d1b195e6d6ce9d5ee61bd023aeb61197187807442501b3e24b68b87ff6: Status 404 returned error can't find the container with id f7a6e0d1b195e6d6ce9d5ee61bd023aeb61197187807442501b3e24b68b87ff6
Sep 09 05:49:35.558273 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.558249    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xlfg6"
Sep 09 05:49:35.559301 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:35.559279    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f7dac2_803f_4c17_a41f_af2157f1670e.slice/crio-850092813f28c72cdc8cbf0a13e6f6ed31d11a1fbf9858fcf0501e86662fbc2f WatchSource:0}: Error finding container 850092813f28c72cdc8cbf0a13e6f6ed31d11a1fbf9858fcf0501e86662fbc2f: Status 404 returned error can't find the container with id 850092813f28c72cdc8cbf0a13e6f6ed31d11a1fbf9858fcf0501e86662fbc2f
Sep 09 05:49:35.565878 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.565846    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v"
Sep 09 05:49:35.568323 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:35.568300    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4fc8ea_c8ee_42aa_b597_b3eb70edb43c.slice/crio-15967fd6eaeae8ba53337baf249065cf70650e7116472ec2cb91bf9c0713b8ab WatchSource:0}: Error finding container 15967fd6eaeae8ba53337baf249065cf70650e7116472ec2cb91bf9c0713b8ab: Status 404 returned error can't find the container with id 15967fd6eaeae8ba53337baf249065cf70650e7116472ec2cb91bf9c0713b8ab
Sep 09 05:49:35.577790 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:49:35.577769    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fcf3f6_42e5_4af7_9b85_cb0f1935d39c.slice/crio-362538689b20a26ef976a25401ab69b688d96c5457c75e799e623c613b0af407 WatchSource:0}: Error finding container 362538689b20a26ef976a25401ab69b688d96c5457c75e799e623c613b0af407: Status 404 returned error can't find the container with id 362538689b20a26ef976a25401ab69b688d96c5457c75e799e623c613b0af407
Sep 09 05:49:35.669646 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.669616    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 05:49:35.813499 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.813409    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:35.813499 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.813453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs\") pod \"network-metrics-daemon-qgb26\" (UID: \"910ab0f9-8007-4ec5-b5b9-bf0e32c85593\") " pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:35.813709 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.813602    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:35.813709 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.813613    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:35.813709 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.813678    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret podName:9e1d1039-4eb0-4691-bd7f-21dfdd3870ca nodeName:}" failed. No retries permitted until 2026-09-09 05:49:36.813657737 +0000 UTC m=+4.029987629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret") pod "global-pull-secret-syncer-h8x9t" (UID: "9e1d1039-4eb0-4691-bd7f-21dfdd3870ca") : object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:35.813709 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.813703    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs podName:910ab0f9-8007-4ec5-b5b9-bf0e32c85593 nodeName:}" failed. No retries permitted until 2026-09-09 05:49:36.813692392 +0000 UTC m=+4.030022308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs") pod "network-metrics-daemon-qgb26" (UID: "910ab0f9-8007-4ec5-b5b9-bf0e32c85593") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:35.914215 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:35.914186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrqj\" (UniqueName: \"kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj\") pod \"network-check-target-zkc2s\" (UID: \"4fb7e960-b0a8-4ba1-88b7-616e4910732a\") " pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:35.914364 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.914347    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 05:49:35.914412 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.914365    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 05:49:35.914412 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.914375    2564 projected.go:196] Error preparing data for projected volume kube-api-access-zkrqj for pod openshift-network-diagnostics/network-check-target-zkc2s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:35.914481 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:35.914429    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj podName:4fb7e960-b0a8-4ba1-88b7-616e4910732a nodeName:}" failed. No retries permitted until 2026-09-09 05:49:36.914414133 +0000 UTC m=+4.130743997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkrqj" (UniqueName: "kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj") pod "network-check-target-zkc2s" (UID: "4fb7e960-b0a8-4ba1-88b7-616e4910732a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:36.271150 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:36.271106    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 05:44:35 +0000 UTC" deadline="2028-06-14 10:56:37.19371783 +0000 UTC"
Sep 09 05:49:36.271150 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:36.271145    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15461h7m0.922577055s"
Sep 09 05:49:36.307465 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:36.307428    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v" event={"ID":"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c","Type":"ContainerStarted","Data":"362538689b20a26ef976a25401ab69b688d96c5457c75e799e623c613b0af407"}
Sep 09 05:49:36.308863 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:36.308836    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h8v7x" event={"ID":"40030ec2-6b02-4d13-b794-aa551a09290e","Type":"ContainerStarted","Data":"f7a6e0d1b195e6d6ce9d5ee61bd023aeb61197187807442501b3e24b68b87ff6"}
Sep 09 05:49:36.310143 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:36.310116    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rjd2g" event={"ID":"8199db41-6f04-4d2e-a6ac-2d51fff9ab42","Type":"ContainerStarted","Data":"6cba0664fc2cb03b10d8b29c6c458f642c49a3c1e084092ceed41f5cc3b773f2"}
Sep 09 05:49:36.311614 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:36.311589    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tz58z" event={"ID":"3c67f7d9-3ba8-43ad-bc02-a063ab1f8c91","Type":"ContainerStarted","Data":"61ec9d33b82d8eb40a20938241b7656f338c0fa044c6d41a678c26577179aa69"}
Sep 09 05:49:36.312817 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:36.312779    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xlfg6" event={"ID":"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c","Type":"ContainerStarted","Data":"15967fd6eaeae8ba53337baf249065cf70650e7116472ec2cb91bf9c0713b8ab"}
Sep 09 05:49:36.314161 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:36.314122    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb" event={"ID":"46f7dac2-803f-4c17-a41f-af2157f1670e","Type":"ContainerStarted","Data":"850092813f28c72cdc8cbf0a13e6f6ed31d11a1fbf9858fcf0501e86662fbc2f"}
Sep 09 05:49:36.315471 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:36.315440    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dmx8r" event={"ID":"f4234e30-8f0b-47d1-a67d-36deddc3a89c","Type":"ContainerStarted","Data":"d8260f8e71da9c03d935a94134eabd85be12f52a50babb3b6d4da3966b27e243"}
Sep 09 05:49:36.316745 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:36.316721    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f7clt" event={"ID":"d87a3026-ed95-4af8-b013-fc6640e5ce48","Type":"ContainerStarted","Data":"eedad4ed7c2ee5bc8105e47e70d2ac3939484d3bb2ae853860f08aa5b7fe6dd3"}
Sep 09 05:49:36.320267 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:36.319093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz2h" event={"ID":"9d280b22-a809-4323-8a18-2049d0394312","Type":"ContainerStarted","Data":"2a80cd46a5c621075e1dc1f97fea8e545840c3da147698026a9491b44a8a653d"}
Sep 09 05:49:36.821715 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:36.821237    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:36.821715 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:36.821272    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs\") pod \"network-metrics-daemon-qgb26\" (UID: \"910ab0f9-8007-4ec5-b5b9-bf0e32c85593\") " pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:36.821715 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:36.821357    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:36.821715 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:36.821357    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:36.821715 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:36.821403    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs podName:910ab0f9-8007-4ec5-b5b9-bf0e32c85593 nodeName:}" failed. No retries permitted until 2026-09-09 05:49:38.821386236 +0000 UTC m=+6.037716117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs") pod "network-metrics-daemon-qgb26" (UID: "910ab0f9-8007-4ec5-b5b9-bf0e32c85593") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:36.821715 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:36.821420    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret podName:9e1d1039-4eb0-4691-bd7f-21dfdd3870ca nodeName:}" failed. No retries permitted until 2026-09-09 05:49:38.821411723 +0000 UTC m=+6.037741584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret") pod "global-pull-secret-syncer-h8x9t" (UID: "9e1d1039-4eb0-4691-bd7f-21dfdd3870ca") : object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:36.826059 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:36.826036    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 05:49:36.922423 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:36.922387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrqj\" (UniqueName: \"kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj\") pod \"network-check-target-zkc2s\" (UID: \"4fb7e960-b0a8-4ba1-88b7-616e4910732a\") " pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:36.922613 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:36.922582    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 05:49:36.922613 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:36.922607    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 05:49:36.922725 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:36.922622    2564 projected.go:196] Error preparing data for projected volume kube-api-access-zkrqj for pod openshift-network-diagnostics/network-check-target-zkc2s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:36.922725 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:36.922682    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj podName:4fb7e960-b0a8-4ba1-88b7-616e4910732a nodeName:}" failed. No retries permitted until 2026-09-09 05:49:38.922664546 +0000 UTC m=+6.138994419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkrqj" (UniqueName: "kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj") pod "network-check-target-zkc2s" (UID: "4fb7e960-b0a8-4ba1-88b7-616e4910732a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:37.271665 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:37.271479    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 05:44:35 +0000 UTC" deadline="2028-04-19 14:48:36.199014224 +0000 UTC"
Sep 09 05:49:37.271665 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:37.271551    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14120h58m58.927472555s"
Sep 09 05:49:37.298322 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:37.297611    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:37.298322 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:37.297745    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:49:37.298322 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:37.298138    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:37.298322 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:37.298231    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:49:37.298660 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:37.298546    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:37.298722 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:37.298663    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:49:38.838839 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:38.838148    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:38.838839 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:38.838202    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs\") pod \"network-metrics-daemon-qgb26\" (UID: \"910ab0f9-8007-4ec5-b5b9-bf0e32c85593\") " pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:38.838839 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:38.838314    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:38.838839 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:38.838376    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs podName:910ab0f9-8007-4ec5-b5b9-bf0e32c85593 nodeName:}" failed. No retries permitted until 2026-09-09 05:49:42.838355351 +0000 UTC m=+10.054685237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs") pod "network-metrics-daemon-qgb26" (UID: "910ab0f9-8007-4ec5-b5b9-bf0e32c85593") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:38.838839 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:38.838759    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:38.838839 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:38.838809    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret podName:9e1d1039-4eb0-4691-bd7f-21dfdd3870ca nodeName:}" failed. No retries permitted until 2026-09-09 05:49:42.838794979 +0000 UTC m=+10.055124846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret") pod "global-pull-secret-syncer-h8x9t" (UID: "9e1d1039-4eb0-4691-bd7f-21dfdd3870ca") : object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:38.938911 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:38.938870    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrqj\" (UniqueName: \"kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj\") pod \"network-check-target-zkc2s\" (UID: \"4fb7e960-b0a8-4ba1-88b7-616e4910732a\") " pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:38.939095 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:38.939039    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 05:49:38.939095 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:38.939062    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 05:49:38.939095 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:38.939075    2564 projected.go:196] Error preparing data for projected volume kube-api-access-zkrqj for pod openshift-network-diagnostics/network-check-target-zkc2s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:38.939246 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:38.939137    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj podName:4fb7e960-b0a8-4ba1-88b7-616e4910732a nodeName:}" failed. No retries permitted until 2026-09-09 05:49:42.939118327 +0000 UTC m=+10.155448194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkrqj" (UniqueName: "kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj") pod "network-check-target-zkc2s" (UID: "4fb7e960-b0a8-4ba1-88b7-616e4910732a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:39.296791 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:39.296668    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:39.296791 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:39.296712    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:39.296999 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:39.296810    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:49:39.297158 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:39.297115    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:39.297246 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:39.297212    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:49:39.297302 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:39.297287    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:49:41.296319 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:41.296238    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:41.296734 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:41.296238    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:41.296734 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:41.296379    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:49:41.296734 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:41.296462    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:49:41.296734 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:41.296245    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:41.296734 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:41.296579    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:49:42.875903 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:42.875860    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:42.876571 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:42.875918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs\") pod \"network-metrics-daemon-qgb26\" (UID: \"910ab0f9-8007-4ec5-b5b9-bf0e32c85593\") " pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:42.876571 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:42.876045    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:42.876571 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:42.876050    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:42.876571 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:42.876106    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs podName:910ab0f9-8007-4ec5-b5b9-bf0e32c85593 nodeName:}" failed. No retries permitted until 2026-09-09 05:49:50.87608705 +0000 UTC m=+18.092416915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs") pod "network-metrics-daemon-qgb26" (UID: "910ab0f9-8007-4ec5-b5b9-bf0e32c85593") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:42.876571 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:42.876124    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret podName:9e1d1039-4eb0-4691-bd7f-21dfdd3870ca nodeName:}" failed. No retries permitted until 2026-09-09 05:49:50.876114935 +0000 UTC m=+18.092444800 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret") pod "global-pull-secret-syncer-h8x9t" (UID: "9e1d1039-4eb0-4691-bd7f-21dfdd3870ca") : object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:42.977061 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:42.977025    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrqj\" (UniqueName: \"kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj\") pod \"network-check-target-zkc2s\" (UID: \"4fb7e960-b0a8-4ba1-88b7-616e4910732a\") " pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:42.977243 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:42.977070    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 05:49:42.977243 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:42.977088    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 05:49:42.977243 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:42.977099    2564 projected.go:196] Error preparing data for projected volume kube-api-access-zkrqj for pod openshift-network-diagnostics/network-check-target-zkc2s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:42.977243 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:42.977170    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj podName:4fb7e960-b0a8-4ba1-88b7-616e4910732a nodeName:}" failed. No retries permitted until 2026-09-09 05:49:50.977151244 +0000 UTC m=+18.193481121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkrqj" (UniqueName: "kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj") pod "network-check-target-zkc2s" (UID: "4fb7e960-b0a8-4ba1-88b7-616e4910732a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:43.296504 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:43.296406    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:43.296667 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:43.296533    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:49:43.296811 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:43.296784    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:43.296952 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:43.296917    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:43.297022 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:43.296914    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:49:43.297077 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:43.297045    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:49:45.296105 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:45.296074    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:45.296535 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:45.296115    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:45.296535 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:45.296071    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:45.296535 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:45.296185    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:49:45.296535 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:45.296317    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:49:45.296535 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:45.296438    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:49:47.296671 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:47.296638    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:47.297161 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:47.296639    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:47.297161 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:47.296770    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:49:47.297161 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:47.296789    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:47.297161 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:47.296907    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:49:47.297161 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:47.296995    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:49:49.296771 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:49.296733    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:49.297341 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:49.296733    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:49.297341 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:49.296861    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:49:49.297341 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:49.296733    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:49.297341 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:49.296950    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:49:49.297341 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:49.297047    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:49:50.939705 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:50.939650    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:50.940190 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:50.939726    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs\") pod \"network-metrics-daemon-qgb26\" (UID: \"910ab0f9-8007-4ec5-b5b9-bf0e32c85593\") " pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:50.940190 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:50.939823    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:50.940190 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:50.939904    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret podName:9e1d1039-4eb0-4691-bd7f-21dfdd3870ca nodeName:}" failed. No retries permitted until 2026-09-09 05:50:06.939883405 +0000 UTC m=+34.156213287 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret") pod "global-pull-secret-syncer-h8x9t" (UID: "9e1d1039-4eb0-4691-bd7f-21dfdd3870ca") : object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:50.940190 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:50.939829    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:50.940190 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:50.939976    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs podName:910ab0f9-8007-4ec5-b5b9-bf0e32c85593 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:06.939959191 +0000 UTC m=+34.156289068 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs") pod "network-metrics-daemon-qgb26" (UID: "910ab0f9-8007-4ec5-b5b9-bf0e32c85593") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:51.040778 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:51.040738    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrqj\" (UniqueName: \"kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj\") pod \"network-check-target-zkc2s\" (UID: \"4fb7e960-b0a8-4ba1-88b7-616e4910732a\") " pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:51.040941 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:51.040889    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 05:49:51.040941 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:51.040905    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 05:49:51.040941 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:51.040918    2564 projected.go:196] Error preparing data for projected volume kube-api-access-zkrqj for pod openshift-network-diagnostics/network-check-target-zkc2s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:51.041050 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:51.040985    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj podName:4fb7e960-b0a8-4ba1-88b7-616e4910732a nodeName:}" failed. No retries permitted until 2026-09-09 05:50:07.04096799 +0000 UTC m=+34.257297859 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkrqj" (UniqueName: "kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj") pod "network-check-target-zkc2s" (UID: "4fb7e960-b0a8-4ba1-88b7-616e4910732a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:51.296022 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:51.295953    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:51.296175 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:51.295953    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:51.296175 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:51.296062    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:49:51.296175 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:51.295953    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:51.296340 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:51.296131    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:49:51.296340 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:51.296254    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:49:53.297346 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:53.297306    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:53.297794 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:53.297405    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:49:53.297794 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:53.297522    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:53.297794 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:53.297554    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:53.297794 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:53.297629    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:49:53.297794 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:53.297728    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:49:54.364955 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.364602    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb" event={"ID":"46f7dac2-803f-4c17-a41f-af2157f1670e","Type":"ContainerStarted","Data":"7ddaf8e02764c0bf69526a3dd1e9936675f67a8fbf509ee91cd6b4c3d07ae136"}
Sep 09 05:49:54.365613 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.365055    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb" event={"ID":"46f7dac2-803f-4c17-a41f-af2157f1670e","Type":"ContainerStarted","Data":"bc3f5d016035d8a728420213868fbd017086c301b3a4029fa66fa6320881b858"}
Sep 09 05:49:54.365613 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.365101    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb" event={"ID":"46f7dac2-803f-4c17-a41f-af2157f1670e","Type":"ContainerStarted","Data":"00f2c50454906de4e611a0d1af0658bf2cc392fb54e01fcfdde8bbfc8f9f1895"}
Sep 09 05:49:54.365613 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.365129    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb" event={"ID":"46f7dac2-803f-4c17-a41f-af2157f1670e","Type":"ContainerStarted","Data":"27f31c302c6b6816012ebd306a05aebe3898f66a4f1240e19a246feb1e2b766f"}
Sep 09 05:49:54.365613 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.365157    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb" event={"ID":"46f7dac2-803f-4c17-a41f-af2157f1670e","Type":"ContainerStarted","Data":"3150d1d546685ba87a060ccb0a7014b1fa3bfc1c9264bdf333a7e9882538852a"}
Sep 09 05:49:54.365613 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.365185    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb" event={"ID":"46f7dac2-803f-4c17-a41f-af2157f1670e","Type":"ContainerStarted","Data":"7d8053b7a4c5b10f5c313d7f5ae490ed4f04cf1b3c2ba1b34a2e85cce0baa503"}
Sep 09 05:49:54.366134 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.366103    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dmx8r" event={"ID":"f4234e30-8f0b-47d1-a67d-36deddc3a89c","Type":"ContainerStarted","Data":"2c9f1065377dcd45a8e33ce8b4d0b05879faeb8e9bedfc2badf28e8f44331b20"}
Sep 09 05:49:54.367471 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.367452    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f7clt" event={"ID":"d87a3026-ed95-4af8-b013-fc6640e5ce48","Type":"ContainerStarted","Data":"ab5a787d66539467f23f00655ccf3a3842f633f2723e2cf0eb6c13bc20f1a12c"}
Sep 09 05:49:54.368822 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.368799    2564 generic.go:356] "Generic (PLEG): container finished" podID="9d280b22-a809-4323-8a18-2049d0394312" containerID="41b924bb83917279504e9428c4f5436285d70d6057a1bf09b894e4a44a0349f4" exitCode=0
Sep 09 05:49:54.368901 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.368878    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz2h" event={"ID":"9d280b22-a809-4323-8a18-2049d0394312","Type":"ContainerDied","Data":"41b924bb83917279504e9428c4f5436285d70d6057a1bf09b894e4a44a0349f4"}
Sep 09 05:49:54.370477 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.370455    2564 generic.go:356] "Generic (PLEG): container finished" podID="1fb08136825eb2ae62fa73ecf8896a9e" containerID="58f4fe8a44af9db2e6533bfcbcde0d2499af23d9324958bb8c041cb7ad1dfe3f" exitCode=0
Sep 09 05:49:54.370545 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.370519    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal" event={"ID":"1fb08136825eb2ae62fa73ecf8896a9e","Type":"ContainerDied","Data":"58f4fe8a44af9db2e6533bfcbcde0d2499af23d9324958bb8c041cb7ad1dfe3f"}
Sep 09 05:49:54.371943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.371903    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v" event={"ID":"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c","Type":"ContainerStarted","Data":"5f8053cb8b3777deb06cbaa7a78a4832af504b6a77f14d2b6df6f13ad56db68e"}
Sep 09 05:49:54.373377 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.373358    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h8v7x" event={"ID":"40030ec2-6b02-4d13-b794-aa551a09290e","Type":"ContainerStarted","Data":"4f2a301e71f9afc8cc779b871fbccdae680661408e53fc87cdf45df6b15882dc"}
Sep 09 05:49:54.374650 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.374628    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tz58z" event={"ID":"3c67f7d9-3ba8-43ad-bc02-a063ab1f8c91","Type":"ContainerStarted","Data":"b54a2f18b476b9e0481443bbbd79522eb663e49abd2a73635c6e7d86e1e4c4ad"}
Sep 09 05:49:54.375935 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.375918    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal" event={"ID":"b918f93a5380fe0ffc07a388ced93b00","Type":"ContainerStarted","Data":"4e984c21539851b5f5f88683574ce323a4e9d4852dfb1ad95e74e1a504106c97"}
Sep 09 05:49:54.377171 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.377154    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xlfg6" event={"ID":"0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c","Type":"ContainerStarted","Data":"9439d4becc4cd075cf4723b14edec7a8892b5f4fd032891a1b40addabb581506"}
Sep 09 05:49:54.395862 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.395816    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f7clt" podStartSLOduration=3.407221103 podStartE2EDuration="21.395802082s" totalImagesPullingTime="17.988580979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:54.395548898 +0000 UTC m=+21.611878773" watchObservedRunningTime="2026-09-09 05:49:54.395802082 +0000 UTC m=+21.612131965"
Sep 09 05:49:54.395991 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.395935    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dmx8r" podStartSLOduration=3.811498715 podStartE2EDuration="21.395929013s" totalImagesPullingTime="17.584430298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:54.380812272 +0000 UTC m=+21.597142155" watchObservedRunningTime="2026-09-09 05:49:54.395929013 +0000 UTC m=+21.612258901"
Sep 09 05:49:54.411658 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.411619    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tz58z" podStartSLOduration=3.434149374 podStartE2EDuration="21.411604177s" totalImagesPullingTime="17.977454803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:54.41079608 +0000 UTC m=+21.627125963" watchObservedRunningTime="2026-09-09 05:49:54.411604177 +0000 UTC m=+21.627934061"
Sep 09 05:49:54.430683 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.430643    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h8v7x" podStartSLOduration=11.240137658 podStartE2EDuration="21.430626598s" totalImagesPullingTime="10.19048894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:54.430542652 +0000 UTC m=+21.646872536" watchObservedRunningTime="2026-09-09 05:49:54.430626598 +0000 UTC m=+21.646956482"
Sep 09 05:49:54.454603 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.454567    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xlfg6" podStartSLOduration=3.454463433 podStartE2EDuration="21.454554045s" totalImagesPullingTime="18.000090612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:54.454283796 +0000 UTC m=+21.670613680" watchObservedRunningTime="2026-09-09 05:49:54.454554045 +0000 UTC m=+21.670883928"
Sep 09 05:49:54.490345 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:54.490306    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.ec2.internal" podStartSLOduration=20.490292299 podStartE2EDuration="20.490292299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:54.469211319 +0000 UTC m=+21.685541204" watchObservedRunningTime="2026-09-09 05:49:54.490292299 +0000 UTC m=+21.706622182"
Sep 09 05:49:55.296129 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:55.296054    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:55.296325 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:55.296066    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:55.296325 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:55.296197    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:49:55.296325 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:55.296244    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:55.296505 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:55.296422    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:49:55.296568 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:55.296511    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:49:55.381482 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:55.381424    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rjd2g" event={"ID":"8199db41-6f04-4d2e-a6ac-2d51fff9ab42","Type":"ContainerStarted","Data":"26bd8cdae2947bbf05ca55797d57bd48347ef0225e941cdba35383f1908d5642"}
Sep 09 05:49:55.396153 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:55.396068    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rjd2g" podStartSLOduration=4.806409252 podStartE2EDuration="22.39603819s" totalImagesPullingTime="17.589628938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:55.39571658 +0000 UTC m=+22.612046465" watchObservedRunningTime="2026-09-09 05:49:55.39603819 +0000 UTC m=+22.612368074"
Sep 09 05:49:55.584572 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:55.584547    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tz58z"
Sep 09 05:49:55.585123 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:55.585093    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tz58z"
Sep 09 05:49:55.659467 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:55.659443    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 05:49:56.381310 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:56.381205    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T05:49:55.659463162Z","UUID":"ed33b8f7-6c01-4b4b-8390-257c7d921bb3","Handler":null,"Name":"","Endpoint":""}
Sep 09 05:49:56.383148 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:56.383125    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 05:49:56.383682 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:56.383162    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 05:49:56.393200 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:56.393173    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb" event={"ID":"46f7dac2-803f-4c17-a41f-af2157f1670e","Type":"ContainerStarted","Data":"98991a92a55648396d59214ebd1993772e8a7548faacf7f52c72c0284d7dcbad"}
Sep 09 05:49:56.396406 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:56.396376    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal" event={"ID":"1fb08136825eb2ae62fa73ecf8896a9e","Type":"ContainerStarted","Data":"60fd68ac94d6d4951c2452ed1807433485cb86aa469cfd9f0ee1d2d5dc274c1a"}
Sep 09 05:49:56.401274 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:56.400479    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v" event={"ID":"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c","Type":"ContainerStarted","Data":"70299942c69baf5735fd52476f3e865e1e3bcc982e45873a4d27fd509dd46509"}
Sep 09 05:49:56.401274 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:56.401121    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tz58z"
Sep 09 05:49:56.401850 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:56.401807    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tz58z"
Sep 09 05:49:56.433026 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:56.432974    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.ec2.internal" podStartSLOduration=22.432953692 podStartE2EDuration="22.432953692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:56.418210629 +0000 UTC m=+23.634540513" watchObservedRunningTime="2026-09-09 05:49:56.432953692 +0000 UTC m=+23.649283577"
Sep 09 05:49:57.297064 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:57.296859    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:57.297246 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:57.296869    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:57.297246 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:57.297206    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:49:57.297246 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:57.296869    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:57.297375 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:57.297264    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:49:57.297375 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:57.297350    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:49:57.404870 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:57.404829    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v" event={"ID":"44fcf3f6-42e5-4af7-9b85-cb0f1935d39c","Type":"ContainerStarted","Data":"209370d6591de83363b84a8a66fdd28439272aaf599e970dafb48823f722fe59"}
Sep 09 05:49:57.422486 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:57.422335    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdb9v" podStartSLOduration=3.984990822 podStartE2EDuration="24.422321419s" totalImagesPullingTime="20.437330597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:57.421832766 +0000 UTC m=+24.638162650" watchObservedRunningTime="2026-09-09 05:49:57.422321419 +0000 UTC m=+24.638651303"
Sep 09 05:49:59.295832 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:59.295744    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:49:59.295832 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:59.295757    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:49:59.295832 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:59.295757    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:49:59.296263 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:59.295866    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:49:59.296263 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:59.295939    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:49:59.296263 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:49:59.296023    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:49:59.412894 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:59.412857    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb" event={"ID":"46f7dac2-803f-4c17-a41f-af2157f1670e","Type":"ContainerStarted","Data":"953697d359cecae634b9423fc71e5a1014f9f2d32ce07648737aafed22f5ff5f"}
Sep 09 05:49:59.413132 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:59.413108    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:59.428055 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:59.428025    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:49:59.443611 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:49:59.443567    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb" podStartSLOduration=8.364942791 podStartE2EDuration="26.443551662s" totalImagesPullingTime="18.078608871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:59.443362206 +0000 UTC m=+26.659692101" watchObservedRunningTime="2026-09-09 05:49:59.443551662 +0000 UTC m=+26.659881546"
Sep 09 05:50:00.417455 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:00.417206    2564 generic.go:356] "Generic (PLEG): container finished" podID="9d280b22-a809-4323-8a18-2049d0394312" containerID="cdbbd24f05877fc0921cb3bceffaf67d3e57928d2b3b9f22083f05566ad0d124" exitCode=0
Sep 09 05:50:00.417933 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:00.417263    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz2h" event={"ID":"9d280b22-a809-4323-8a18-2049d0394312","Type":"ContainerDied","Data":"cdbbd24f05877fc0921cb3bceffaf67d3e57928d2b3b9f22083f05566ad0d124"}
Sep 09 05:50:00.417933 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:00.417611    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 05:50:00.418109 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:00.418059    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:50:00.432995 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:00.432975    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:50:01.186824 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:01.186789    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h8x9t"]
Sep 09 05:50:01.186971 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:01.186924    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:50:01.187062 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:01.187039    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:50:01.189651 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:01.189631    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qgb26"]
Sep 09 05:50:01.189758 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:01.189743    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:50:01.189875 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:01.189858    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:50:01.190256 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:01.190233    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zkc2s"]
Sep 09 05:50:01.190345 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:01.190332    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:50:01.190548 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:01.190412    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:50:01.422156 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:01.422077    2564 generic.go:356] "Generic (PLEG): container finished" podID="9d280b22-a809-4323-8a18-2049d0394312" containerID="b06234984b9c0a613d07dabb57beda92ea24d7a6e295b29032eabbe961a785ae" exitCode=0
Sep 09 05:50:01.422720 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:01.422157    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz2h" event={"ID":"9d280b22-a809-4323-8a18-2049d0394312","Type":"ContainerDied","Data":"b06234984b9c0a613d07dabb57beda92ea24d7a6e295b29032eabbe961a785ae"}
Sep 09 05:50:01.422720 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:01.422257    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 05:50:02.296707 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:02.296670    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:50:02.296877 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:02.296788    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:50:02.427836 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:02.427744    2564 generic.go:356] "Generic (PLEG): container finished" podID="9d280b22-a809-4323-8a18-2049d0394312" containerID="8b3304a0d0667ca7b04aca273d458eecb4f63aca86269386836109ce6bc20bcd" exitCode=0
Sep 09 05:50:02.428381 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:02.427831    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz2h" event={"ID":"9d280b22-a809-4323-8a18-2049d0394312","Type":"ContainerDied","Data":"8b3304a0d0667ca7b04aca273d458eecb4f63aca86269386836109ce6bc20bcd"}
Sep 09 05:50:02.428381 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:02.427924    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 05:50:03.298255 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:03.298008    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:50:03.298432 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:03.298075    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:50:03.298432 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:03.298366    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:50:03.298573 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:03.298486    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:50:04.296726 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:04.296683    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:50:04.297316 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:04.296804    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:50:04.888570 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:04.888538    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:50:04.888830 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:04.888788    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 05:50:04.907143 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:04.907096    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb" podUID="46f7dac2-803f-4c17-a41f-af2157f1670e" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 09 05:50:04.916879 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:04.916841    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb" podUID="46f7dac2-803f-4c17-a41f-af2157f1670e" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 09 05:50:05.296189 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:05.296107    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:50:05.296340 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:05.296107    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:50:05.296340 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:05.296256    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:50:05.296340 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:05.296317    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:50:06.296757 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:06.296719    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:50:06.297156 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:06.296838    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkc2s" podUID="4fb7e960-b0a8-4ba1-88b7-616e4910732a"
Sep 09 05:50:06.951480 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:06.951447    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:50:06.951480 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:06.951485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs\") pod \"network-metrics-daemon-qgb26\" (UID: \"910ab0f9-8007-4ec5-b5b9-bf0e32c85593\") " pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:50:06.951685 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:06.951601    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:50:06.951685 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:06.951655    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs podName:910ab0f9-8007-4ec5-b5b9-bf0e32c85593 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:38.951642654 +0000 UTC m=+66.167972516 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs") pod "network-metrics-daemon-qgb26" (UID: "910ab0f9-8007-4ec5-b5b9-bf0e32c85593") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:50:06.951765 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:06.951601    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 05:50:06.951765 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:06.951734    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret podName:9e1d1039-4eb0-4691-bd7f-21dfdd3870ca nodeName:}" failed. No retries permitted until 2026-09-09 05:50:38.951721349 +0000 UTC m=+66.168051224 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret") pod "global-pull-secret-syncer-h8x9t" (UID: "9e1d1039-4eb0-4691-bd7f-21dfdd3870ca") : object "kube-system"/"original-pull-secret" not registered
Sep 09 05:50:07.052713 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.052685    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrqj\" (UniqueName: \"kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj\") pod \"network-check-target-zkc2s\" (UID: \"4fb7e960-b0a8-4ba1-88b7-616e4910732a\") " pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:50:07.052866 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.052847    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 05:50:07.052922 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.052870    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 05:50:07.052922 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.052880    2564 projected.go:196] Error preparing data for projected volume kube-api-access-zkrqj for pod openshift-network-diagnostics/network-check-target-zkc2s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:50:07.053000 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.052928    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj podName:4fb7e960-b0a8-4ba1-88b7-616e4910732a nodeName:}" failed. No retries permitted until 2026-09-09 05:50:39.052915343 +0000 UTC m=+66.269245210 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkrqj" (UniqueName: "kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj") pod "network-check-target-zkc2s" (UID: "4fb7e960-b0a8-4ba1-88b7-616e4910732a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:50:07.295983 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.295955    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:50:07.296053 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.295992    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:50:07.296103 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.296086    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgb26" podUID="910ab0f9-8007-4ec5-b5b9-bf0e32c85593"
Sep 09 05:50:07.296231 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.296214    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8x9t" podUID="9e1d1039-4eb0-4691-bd7f-21dfdd3870ca"
Sep 09 05:50:07.441598 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.441564    2564 generic.go:356] "Generic (PLEG): container finished" podID="9d280b22-a809-4323-8a18-2049d0394312" containerID="b57d11e44b23ef2f0adbc26d0954cf31e7ed17f1ed23f6bbf363579bd7daed47" exitCode=0
Sep 09 05:50:07.441988 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.441607    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz2h" event={"ID":"9d280b22-a809-4323-8a18-2049d0394312","Type":"ContainerDied","Data":"b57d11e44b23ef2f0adbc26d0954cf31e7ed17f1ed23f6bbf363579bd7daed47"}
Sep 09 05:50:07.630961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.630939    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-250.ec2.internal" event="NodeReady"
Sep 09 05:50:07.631082 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.631072    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 05:50:07.663685 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.663658    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64cdf68b7d-mdwmv"]
Sep 09 05:50:07.680878 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.680857    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"]
Sep 09 05:50:07.681011 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.680993    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.684105 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.683800    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 05:50:07.684688 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.684668    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gpg8k\""
Sep 09 05:50:07.686019 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.685842    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 05:50:07.686277 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.686195    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 05:50:07.689482 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.689458    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 05:50:07.698991 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.698904    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-78695b457d-lfsxh"]
Sep 09 05:50:07.698991 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.698986    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:07.702548 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.702461    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 05:50:07.702637 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.702560    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 05:50:07.703579 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.703561    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 05:50:07.703705 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.703578    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 05:50:07.703705 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.703621    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8zxvp\""
Sep 09 05:50:07.713853 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.713832    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-7lfm4"]
Sep 09 05:50:07.713972 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.713939    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:07.716231 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.716209    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 05:50:07.716355 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.716341    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 05:50:07.716460 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.716440    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 05:50:07.716616 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.716595    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 05:50:07.717002 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.716982    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-j8bw9\""
Sep 09 05:50:07.717184 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.716541    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 05:50:07.717184 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.717161    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 05:50:07.727880 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.727860    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qb2nd"]
Sep 09 05:50:07.728045 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.728026    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4"
Sep 09 05:50:07.730080 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.730063    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-96lk8\""
Sep 09 05:50:07.730181 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.730066    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 05:50:07.730262 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.730245    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 05:50:07.743448 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.743428    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"]
Sep 09 05:50:07.743565 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.743537    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qb2nd"
Sep 09 05:50:07.745765 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.745742    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 05:50:07.745900 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.745885    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 05:50:07.745971 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.745939    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-p7drt\""
Sep 09 05:50:07.757573 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.757555    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4xrn\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-kube-api-access-c4xrn\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.757672 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.757594    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-certificates\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.757672 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.757617    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-bound-sa-token\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.757672 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.757633    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/48cf9622-0adf-4129-955f-3e82431ab4ab-trusted-ca\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.757790 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.757687    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/48cf9622-0adf-4129-955f-3e82431ab4ab-image-registry-private-configuration\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.757790 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.757730    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.757790 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.757759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/48cf9622-0adf-4129-955f-3e82431ab4ab-ca-trust-extracted\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.757877 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.757788    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/48cf9622-0adf-4129-955f-3e82431ab4ab-installation-pull-secrets\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.759580 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.759567    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp"]
Sep 09 05:50:07.759673 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.759657    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"
Sep 09 05:50:07.761891 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.761869    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 05:50:07.761891 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.761884    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 05:50:07.762008 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.761948    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6rzjk\""
Sep 09 05:50:07.762008 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.761978    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 05:50:07.775742 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.775723    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp"
Sep 09 05:50:07.775742 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.775724    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qb2nd"]
Sep 09 05:50:07.775883 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.775777    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"]
Sep 09 05:50:07.775883 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.775803    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64cdf68b7d-mdwmv"]
Sep 09 05:50:07.775883 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.775837    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp"]
Sep 09 05:50:07.775883 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.775859    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78695b457d-lfsxh"]
Sep 09 05:50:07.775883 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.775881    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm"]
Sep 09 05:50:07.778200 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.778180    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 05:50:07.778268 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.778197    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 05:50:07.778268 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.778260    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 05:50:07.778361 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.778199    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 05:50:07.778539 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.778523    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gg67g\""
Sep 09 05:50:07.791687 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.791663    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-5mbcw"]
Sep 09 05:50:07.791780 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.791768    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm"
Sep 09 05:50:07.794220 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.794203    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 05:50:07.794319 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.794234    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jl94f\""
Sep 09 05:50:07.794383 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.794256    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 05:50:07.794383 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.794263    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 05:50:07.794484 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.794272    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 05:50:07.809861 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.809842    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-4wv9c"]
Sep 09 05:50:07.809982 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.809968    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:07.812330 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.812311    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 05:50:07.812428 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.812358    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 05:50:07.812428 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.812358    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-l6l5f\""
Sep 09 05:50:07.812428 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.812401    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 05:50:07.812428 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.812408    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 05:50:07.817895 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.817879    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 05:50:07.831990 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.831968    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-cflnz"]
Sep 09 05:50:07.832098 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.832076    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-4wv9c"
Sep 09 05:50:07.834793 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.834777    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 05:50:07.835084 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.835070    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 05:50:07.835176 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.835150    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-njqfg\""
Sep 09 05:50:07.856102 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.856080    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-7lfm4"]
Sep 09 05:50:07.856209 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.856109    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"]
Sep 09 05:50:07.856209 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.856125    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-4wv9c"]
Sep 09 05:50:07.856209 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.856136    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm"]
Sep 09 05:50:07.856209 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.856149    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-cflnz"]
Sep 09 05:50:07.856209 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.856167    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9gc29"]
Sep 09 05:50:07.856209 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.856194    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:07.858229 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858211    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xrn\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-kube-api-access-c4xrn\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.858314 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858236    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2ce177a-74d8-4240-8eb1-bf8c703e59d6-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rghsp\" (UID: \"c2ce177a-74d8-4240-8eb1-bf8c703e59d6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp"
Sep 09 05:50:07.858314 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858256    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59sn\" (UniqueName: \"kubernetes.io/projected/c2ce177a-74d8-4240-8eb1-bf8c703e59d6-kube-api-access-f59sn\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rghsp\" (UID: \"c2ce177a-74d8-4240-8eb1-bf8c703e59d6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp"
Sep 09 05:50:07.858314 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858296    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2ce177a-74d8-4240-8eb1-bf8c703e59d6-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rghsp\" (UID: \"c2ce177a-74d8-4240-8eb1-bf8c703e59d6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp"
Sep 09 05:50:07.858457 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858326    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-default-certificate\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:07.858457 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfq25\" (UniqueName: \"kubernetes.io/projected/cf0f9476-232d-4dfb-8b60-175fba9847db-kube-api-access-qfq25\") pod \"cluster-samples-operator-7cdcfcf76c-jbrn2\" (UID: \"cf0f9476-232d-4dfb-8b60-175fba9847db\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"
Sep 09 05:50:07.858457 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858385    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-certificates\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.858457 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858402    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-prgks\" (UID: \"f04421dd-ed42-46c3-997e-9bf3d4359d7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:07.858457 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858408    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 05:50:07.858457 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858449    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vktdt\" (UniqueName: \"kubernetes.io/projected/f04421dd-ed42-46c3-997e-9bf3d4359d7a-kube-api-access-vktdt\") pod \"cluster-monitoring-operator-6cdcc96965-prgks\" (UID: \"f04421dd-ed42-46c3-997e-9bf3d4359d7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:07.858727 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858475    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f163cf03-e312-4296-8fd6-cc95260b51b3-nginx-conf\") pod \"networking-console-plugin-79977f9844-7lfm4\" (UID: \"f163cf03-e312-4296-8fd6-cc95260b51b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4"
Sep 09 05:50:07.858727 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858529    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-bound-sa-token\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.858727 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858576    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:07.858727 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858602    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/48cf9622-0adf-4129-955f-3e82431ab4ab-trusted-ca\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.858727 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858651    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/48cf9622-0adf-4129-955f-3e82431ab4ab-image-registry-private-configuration\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.858727 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858692    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 05:50:07.858727 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858702    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.859046 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858742    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 05:50:07.859046 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858744    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/48cf9622-0adf-4129-955f-3e82431ab4ab-ca-trust-extracted\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.859046 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858772    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:07.859046 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858696    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rnksd\""
Sep 09 05:50:07.859046 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858799    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj4lg\" (UniqueName: \"kubernetes.io/projected/be5d31e4-20fc-48c9-935d-087f65b77eca-kube-api-access-qj4lg\") pod \"volume-data-source-validator-695c895bd6-qb2nd\" (UID: \"be5d31e4-20fc-48c9-935d-087f65b77eca\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qb2nd"
Sep 09 05:50:07.859046 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858831    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gpft\" (UniqueName: \"kubernetes.io/projected/08804660-c36f-432a-80c1-6baba82c301f-kube-api-access-6gpft\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:07.859046 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858855    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jbrn2\" (UID: \"cf0f9476-232d-4dfb-8b60-175fba9847db\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"
Sep 09 05:50:07.859046 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858879    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f04421dd-ed42-46c3-997e-9bf3d4359d7a-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-prgks\" (UID: \"f04421dd-ed42-46c3-997e-9bf3d4359d7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:07.859046 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.858910    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/48cf9622-0adf-4129-955f-3e82431ab4ab-installation-pull-secrets\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.859046 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.859024    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 05:50:07.859046 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.859039    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64cdf68b7d-mdwmv: secret "image-registry-tls" not found
Sep 09 05:50:07.859572 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.859100    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls podName:48cf9622-0adf-4129-955f-3e82431ab4ab nodeName:}" failed. No retries permitted until 2026-09-09 05:50:08.359080984 +0000 UTC m=+35.575410852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls") pod "image-registry-64cdf68b7d-mdwmv" (UID: "48cf9622-0adf-4129-955f-3e82431ab4ab") : secret "image-registry-tls" not found
Sep 09 05:50:07.859572 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.859164    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/48cf9622-0adf-4129-955f-3e82431ab4ab-ca-trust-extracted\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.859572 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.859169    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-certificates\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.859572 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.859252    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7lfm4\" (UID: \"f163cf03-e312-4296-8fd6-cc95260b51b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4"
Sep 09 05:50:07.859572 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.859297    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-stats-auth\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:07.859828 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.859655    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 05:50:07.859828 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.859701    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/48cf9622-0adf-4129-955f-3e82431ab4ab-trusted-ca\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.863008 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.862984    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/48cf9622-0adf-4129-955f-3e82431ab4ab-image-registry-private-configuration\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.863193 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.863171    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/48cf9622-0adf-4129-955f-3e82431ab4ab-installation-pull-secrets\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.863896 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.863873    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 05:50:07.865616 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.865589    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-bound-sa-token\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.865807 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.865792    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xrn\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-kube-api-access-c4xrn\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:07.881646 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.881624    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pvjjm"]
Sep 09 05:50:07.881723 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.881715    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9gc29"
Sep 09 05:50:07.884294 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.884273    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lskth\""
Sep 09 05:50:07.884419 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.884329    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 05:50:07.884419 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.884377    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 05:50:07.884419 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.884281    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 05:50:07.899859 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.899839    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-5mbcw"]
Sep 09 05:50:07.899956 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.899875    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9gc29"]
Sep 09 05:50:07.899956 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.899897    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pvjjm"]
Sep 09 05:50:07.900025 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.900003    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:07.902188 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.902170    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 05:50:07.902460 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.902445    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sfr25\""
Sep 09 05:50:07.902553 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.902539    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 05:50:07.960668 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.960636    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-default-certificate\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:07.960800 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.960679    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-prgks\" (UID: \"f04421dd-ed42-46c3-997e-9bf3d4359d7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:07.960800 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.960701    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vktdt\" (UniqueName: \"kubernetes.io/projected/f04421dd-ed42-46c3-997e-9bf3d4359d7a-kube-api-access-vktdt\") pod \"cluster-monitoring-operator-6cdcc96965-prgks\" (UID: \"f04421dd-ed42-46c3-997e-9bf3d4359d7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:07.960800 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.960728    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1b76aac-f159-4940-8a85-64a227c18b05-serving-cert\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:07.960800 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.960752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f163cf03-e312-4296-8fd6-cc95260b51b3-nginx-conf\") pod \"networking-console-plugin-79977f9844-7lfm4\" (UID: \"f163cf03-e312-4296-8fd6-cc95260b51b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4"
Sep 09 05:50:07.960800 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.960786    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skh4f\" (UniqueName: \"kubernetes.io/projected/207d756d-5ccd-456e-bd3f-3ede12b60f78-kube-api-access-skh4f\") pod \"console-operator-85b454874b-5mbcw\" (UID: \"207d756d-5ccd-456e-bd3f-3ede12b60f78\") " pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:07.961046 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.960798    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 05:50:07.961046 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.960816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6cd6c398-b2b7-4c95-8899-18f779265279-serving-cert\") pod \"service-ca-operator-859f94c4b-8hmqm\" (UID: \"6cd6c398-b2b7-4c95-8899-18f779265279\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm"
Sep 09 05:50:07.961046 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.960871    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls podName:f04421dd-ed42-46c3-997e-9bf3d4359d7a nodeName:}" failed. No retries permitted until 2026-09-09 05:50:08.460853424 +0000 UTC m=+35.677183301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-prgks" (UID: "f04421dd-ed42-46c3-997e-9bf3d4359d7a") : secret "cluster-monitoring-operator-tls" not found
Sep 09 05:50:07.961046 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.960947    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj4lg\" (UniqueName: \"kubernetes.io/projected/be5d31e4-20fc-48c9-935d-087f65b77eca-kube-api-access-qj4lg\") pod \"volume-data-source-validator-695c895bd6-qb2nd\" (UID: \"be5d31e4-20fc-48c9-935d-087f65b77eca\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qb2nd"
Sep 09 05:50:07.961046 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.960992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jbrn2\" (UID: \"cf0f9476-232d-4dfb-8b60-175fba9847db\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"
Sep 09 05:50:07.961046 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961023    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1b76aac-f159-4940-8a85-64a227c18b05-tmp\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:07.961340 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961050    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/207d756d-5ccd-456e-bd3f-3ede12b60f78-config\") pod \"console-operator-85b454874b-5mbcw\" (UID: \"207d756d-5ccd-456e-bd3f-3ede12b60f78\") " pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:07.961340 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.961085    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 05:50:07.961340 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961107    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6cd6c398-b2b7-4c95-8899-18f779265279-config\") pod \"service-ca-operator-859f94c4b-8hmqm\" (UID: \"6cd6c398-b2b7-4c95-8899-18f779265279\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm"
Sep 09 05:50:07.961340 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.961139    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls podName:cf0f9476-232d-4dfb-8b60-175fba9847db nodeName:}" failed. No retries permitted until 2026-09-09 05:50:08.461122941 +0000 UTC m=+35.677452803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jbrn2" (UID: "cf0f9476-232d-4dfb-8b60-175fba9847db") : secret "samples-operator-tls" not found
Sep 09 05:50:07.961340 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961179    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59sn\" (UniqueName: \"kubernetes.io/projected/c2ce177a-74d8-4240-8eb1-bf8c703e59d6-kube-api-access-f59sn\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rghsp\" (UID: \"c2ce177a-74d8-4240-8eb1-bf8c703e59d6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp"
Sep 09 05:50:07.961340 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961206    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-stats-auth\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:07.961340 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961229    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfq25\" (UniqueName: \"kubernetes.io/projected/cf0f9476-232d-4dfb-8b60-175fba9847db-kube-api-access-qfq25\") pod \"cluster-samples-operator-7cdcfcf76c-jbrn2\" (UID: \"cf0f9476-232d-4dfb-8b60-175fba9847db\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"
Sep 09 05:50:07.961340 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961245    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2ce177a-74d8-4240-8eb1-bf8c703e59d6-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rghsp\" (UID: \"c2ce177a-74d8-4240-8eb1-bf8c703e59d6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp"
Sep 09 05:50:07.961340 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961267    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4g6\" (UniqueName: \"kubernetes.io/projected/ef7a5caf-c270-423f-919e-e7cb1a10a8ff-kube-api-access-gg4g6\") pod \"network-check-source-685456655f-4wv9c\" (UID: \"ef7a5caf-c270-423f-919e-e7cb1a10a8ff\") " pod="openshift-network-diagnostics/network-check-source-685456655f-4wv9c"
Sep 09 05:50:07.961340 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961283    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1b76aac-f159-4940-8a85-64a227c18b05-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:07.961340 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961300    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/207d756d-5ccd-456e-bd3f-3ede12b60f78-trusted-ca\") pod \"console-operator-85b454874b-5mbcw\" (UID: \"207d756d-5ccd-456e-bd3f-3ede12b60f78\") " pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:07.961340 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961329    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/207d756d-5ccd-456e-bd3f-3ede12b60f78-serving-cert\") pod \"console-operator-85b454874b-5mbcw\" (UID: \"207d756d-5ccd-456e-bd3f-3ede12b60f78\") " pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:07.961961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961360    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:07.961961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961386    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d44x4\" (UniqueName: \"kubernetes.io/projected/be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8-kube-api-access-d44x4\") pod \"ingress-canary-9gc29\" (UID: \"be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8\") " pod="openshift-ingress-canary/ingress-canary-9gc29"
Sep 09 05:50:07.961961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961428    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1b76aac-f159-4940-8a85-64a227c18b05-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:07.961961 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.961485    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 05:50:07.961961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961553    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f163cf03-e312-4296-8fd6-cc95260b51b3-nginx-conf\") pod \"networking-console-plugin-79977f9844-7lfm4\" (UID: \"f163cf03-e312-4296-8fd6-cc95260b51b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4"
Sep 09 05:50:07.961961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961562    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8stq\" (UniqueName: \"kubernetes.io/projected/d1b76aac-f159-4940-8a85-64a227c18b05-kube-api-access-s8stq\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:07.961961 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.961564    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs podName:08804660-c36f-432a-80c1-6baba82c301f nodeName:}" failed. No retries permitted until 2026-09-09 05:50:08.461548493 +0000 UTC m=+35.677878373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs") pod "router-default-78695b457d-lfsxh" (UID: "08804660-c36f-432a-80c1-6baba82c301f") : secret "router-metrics-certs-default" not found
Sep 09 05:50:07.961961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f04421dd-ed42-46c3-997e-9bf3d4359d7a-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-prgks\" (UID: \"f04421dd-ed42-46c3-997e-9bf3d4359d7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:07.961961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961665    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d1b76aac-f159-4940-8a85-64a227c18b05-snapshots\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:07.961961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961709    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:07.961961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961753    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gpft\" (UniqueName: \"kubernetes.io/projected/08804660-c36f-432a-80c1-6baba82c301f-kube-api-access-6gpft\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:07.961961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961785    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7lfm4\" (UID: \"f163cf03-e312-4296-8fd6-cc95260b51b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4"
Sep 09 05:50:07.961961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961811    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8-cert\") pod \"ingress-canary-9gc29\" (UID: \"be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8\") " pod="openshift-ingress-canary/ingress-canary-9gc29"
Sep 09 05:50:07.961961 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.961840    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle podName:08804660-c36f-432a-80c1-6baba82c301f nodeName:}" failed. No retries permitted until 2026-09-09 05:50:08.461823961 +0000 UTC m=+35.678153840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle") pod "router-default-78695b457d-lfsxh" (UID: "08804660-c36f-432a-80c1-6baba82c301f") : configmap references non-existent config key: service-ca.crt
Sep 09 05:50:07.961961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961865    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz58p\" (UniqueName: \"kubernetes.io/projected/6cd6c398-b2b7-4c95-8899-18f779265279-kube-api-access-rz58p\") pod \"service-ca-operator-859f94c4b-8hmqm\" (UID: \"6cd6c398-b2b7-4c95-8899-18f779265279\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm"
Sep 09 05:50:07.962566 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.961904    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2ce177a-74d8-4240-8eb1-bf8c703e59d6-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rghsp\" (UID: \"c2ce177a-74d8-4240-8eb1-bf8c703e59d6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp"
Sep 09 05:50:07.962566 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.961913    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 05:50:07.962566 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:07.961970    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert podName:f163cf03-e312-4296-8fd6-cc95260b51b3 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:08.461954842 +0000 UTC m=+35.678284717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-7lfm4" (UID: "f163cf03-e312-4296-8fd6-cc95260b51b3") : secret "networking-console-plugin-cert" not found
Sep 09 05:50:07.962566 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.962395    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f04421dd-ed42-46c3-997e-9bf3d4359d7a-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-prgks\" (UID: \"f04421dd-ed42-46c3-997e-9bf3d4359d7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:07.962566 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.962483    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2ce177a-74d8-4240-8eb1-bf8c703e59d6-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rghsp\" (UID: \"c2ce177a-74d8-4240-8eb1-bf8c703e59d6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp"
Sep 09 05:50:07.963563 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.963543    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-default-certificate\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:07.963612 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.963569    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-stats-auth\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:07.963854 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.963836    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2ce177a-74d8-4240-8eb1-bf8c703e59d6-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rghsp\" (UID: \"c2ce177a-74d8-4240-8eb1-bf8c703e59d6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp"
Sep 09 05:50:07.969421 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.969400    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vktdt\" (UniqueName: \"kubernetes.io/projected/f04421dd-ed42-46c3-997e-9bf3d4359d7a-kube-api-access-vktdt\") pod \"cluster-monitoring-operator-6cdcc96965-prgks\" (UID: \"f04421dd-ed42-46c3-997e-9bf3d4359d7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:07.969554 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.969403    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj4lg\" (UniqueName: \"kubernetes.io/projected/be5d31e4-20fc-48c9-935d-087f65b77eca-kube-api-access-qj4lg\") pod \"volume-data-source-validator-695c895bd6-qb2nd\" (UID: \"be5d31e4-20fc-48c9-935d-087f65b77eca\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qb2nd"
Sep 09 05:50:07.969769 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.969739    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gpft\" (UniqueName: \"kubernetes.io/projected/08804660-c36f-432a-80c1-6baba82c301f-kube-api-access-6gpft\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:07.969769 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.969753    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59sn\" (UniqueName: \"kubernetes.io/projected/c2ce177a-74d8-4240-8eb1-bf8c703e59d6-kube-api-access-f59sn\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rghsp\" (UID: \"c2ce177a-74d8-4240-8eb1-bf8c703e59d6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp"
Sep 09 05:50:07.969932 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:07.969917    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfq25\" (UniqueName: \"kubernetes.io/projected/cf0f9476-232d-4dfb-8b60-175fba9847db-kube-api-access-qfq25\") pod \"cluster-samples-operator-7cdcfcf76c-jbrn2\" (UID: \"cf0f9476-232d-4dfb-8b60-175fba9847db\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"
Sep 09 05:50:08.051766 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.051691    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qb2nd"
Sep 09 05:50:08.063034 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063011    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skh4f\" (UniqueName: \"kubernetes.io/projected/207d756d-5ccd-456e-bd3f-3ede12b60f78-kube-api-access-skh4f\") pod \"console-operator-85b454874b-5mbcw\" (UID: \"207d756d-5ccd-456e-bd3f-3ede12b60f78\") " pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:08.063096 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063047    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6cd6c398-b2b7-4c95-8899-18f779265279-serving-cert\") pod \"service-ca-operator-859f94c4b-8hmqm\" (UID: \"6cd6c398-b2b7-4c95-8899-18f779265279\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm"
Sep 09 05:50:08.063096 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063075    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1b76aac-f159-4940-8a85-64a227c18b05-tmp\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:08.063096 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063091    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/207d756d-5ccd-456e-bd3f-3ede12b60f78-config\") pod \"console-operator-85b454874b-5mbcw\" (UID: \"207d756d-5ccd-456e-bd3f-3ede12b60f78\") " pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:08.063235 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063216    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6cd6c398-b2b7-4c95-8899-18f779265279-config\") pod \"service-ca-operator-859f94c4b-8hmqm\" (UID: \"6cd6c398-b2b7-4c95-8899-18f779265279\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm"
Sep 09 05:50:08.063376 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063357    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4g6\" (UniqueName: \"kubernetes.io/projected/ef7a5caf-c270-423f-919e-e7cb1a10a8ff-kube-api-access-gg4g6\") pod \"network-check-source-685456655f-4wv9c\" (UID: \"ef7a5caf-c270-423f-919e-e7cb1a10a8ff\") " pod="openshift-network-diagnostics/network-check-source-685456655f-4wv9c"
Sep 09 05:50:08.063432 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1b76aac-f159-4940-8a85-64a227c18b05-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:08.063432 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063419    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3f129d-c477-4771-a0cd-4c9e710bb832-config-volume\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:08.063567 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063444    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnzw9\" (UniqueName: \"kubernetes.io/projected/9f3f129d-c477-4771-a0cd-4c9e710bb832-kube-api-access-dnzw9\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:08.063567 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063469    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1b76aac-f159-4940-8a85-64a227c18b05-tmp\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:08.063567 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063475    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/207d756d-5ccd-456e-bd3f-3ede12b60f78-trusted-ca\") pod \"console-operator-85b454874b-5mbcw\" (UID: \"207d756d-5ccd-456e-bd3f-3ede12b60f78\") " pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:08.063714 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063572    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/207d756d-5ccd-456e-bd3f-3ede12b60f78-serving-cert\") pod \"console-operator-85b454874b-5mbcw\" (UID: \"207d756d-5ccd-456e-bd3f-3ede12b60f78\") " pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:08.063714 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063638    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d44x4\" (UniqueName: \"kubernetes.io/projected/be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8-kube-api-access-d44x4\") pod \"ingress-canary-9gc29\" (UID: \"be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8\") " pod="openshift-ingress-canary/ingress-canary-9gc29"
Sep 09 05:50:08.063714 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063674    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1b76aac-f159-4940-8a85-64a227c18b05-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:08.063714 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063699    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8stq\" (UniqueName: \"kubernetes.io/projected/d1b76aac-f159-4940-8a85-64a227c18b05-kube-api-access-s8stq\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:08.064048 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063727    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d1b76aac-f159-4940-8a85-64a227c18b05-snapshots\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:08.064048 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063797    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6cd6c398-b2b7-4c95-8899-18f779265279-config\") pod \"service-ca-operator-859f94c4b-8hmqm\" (UID: \"6cd6c398-b2b7-4c95-8899-18f779265279\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm"
Sep 09 05:50:08.064048 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063807    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8-cert\") pod \"ingress-canary-9gc29\" (UID: \"be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8\") " pod="openshift-ingress-canary/ingress-canary-9gc29"
Sep 09 05:50:08.064048 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063829    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/207d756d-5ccd-456e-bd3f-3ede12b60f78-config\") pod \"console-operator-85b454874b-5mbcw\" (UID: \"207d756d-5ccd-456e-bd3f-3ede12b60f78\") " pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:08.064048 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063941    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1b76aac-f159-4940-8a85-64a227c18b05-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:08.064048 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.063833    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz58p\" (UniqueName: \"kubernetes.io/projected/6cd6c398-b2b7-4c95-8899-18f779265279-kube-api-access-rz58p\") pod \"service-ca-operator-859f94c4b-8hmqm\" (UID: \"6cd6c398-b2b7-4c95-8899-18f779265279\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm"
Sep 09 05:50:08.064048 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.064010    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f3f129d-c477-4771-a0cd-4c9e710bb832-tmp-dir\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:08.064411 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.064081    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1b76aac-f159-4940-8a85-64a227c18b05-serving-cert\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:08.064411 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.064118    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:08.064411 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.064259    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/207d756d-5ccd-456e-bd3f-3ede12b60f78-trusted-ca\") pod \"console-operator-85b454874b-5mbcw\" (UID: \"207d756d-5ccd-456e-bd3f-3ede12b60f78\") " pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:08.064583 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.064480    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d1b76aac-f159-4940-8a85-64a227c18b05-snapshots\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:08.064638 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.064592    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1b76aac-f159-4940-8a85-64a227c18b05-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:08.066003 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.065981    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/207d756d-5ccd-456e-bd3f-3ede12b60f78-serving-cert\") pod \"console-operator-85b454874b-5mbcw\" (UID: \"207d756d-5ccd-456e-bd3f-3ede12b60f78\") " pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:08.066183 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.066160    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1b76aac-f159-4940-8a85-64a227c18b05-serving-cert\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:08.066339 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.066324    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6cd6c398-b2b7-4c95-8899-18f779265279-serving-cert\") pod \"service-ca-operator-859f94c4b-8hmqm\" (UID: \"6cd6c398-b2b7-4c95-8899-18f779265279\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm"
Sep 09 05:50:08.066668 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.066647    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8-cert\") pod \"ingress-canary-9gc29\" (UID: \"be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8\") " pod="openshift-ingress-canary/ingress-canary-9gc29"
Sep 09 05:50:08.071446 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.071419    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skh4f\" (UniqueName: \"kubernetes.io/projected/207d756d-5ccd-456e-bd3f-3ede12b60f78-kube-api-access-skh4f\") pod \"console-operator-85b454874b-5mbcw\" (UID: \"207d756d-5ccd-456e-bd3f-3ede12b60f78\") " pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:08.071585 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.071557    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8stq\" (UniqueName: \"kubernetes.io/projected/d1b76aac-f159-4940-8a85-64a227c18b05-kube-api-access-s8stq\") pod \"insights-operator-76bcd6ccf5-cflnz\" (UID: \"d1b76aac-f159-4940-8a85-64a227c18b05\") " pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:08.071585 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.071579    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4g6\" (UniqueName: \"kubernetes.io/projected/ef7a5caf-c270-423f-919e-e7cb1a10a8ff-kube-api-access-gg4g6\") pod \"network-check-source-685456655f-4wv9c\" (UID: \"ef7a5caf-c270-423f-919e-e7cb1a10a8ff\") " pod="openshift-network-diagnostics/network-check-source-685456655f-4wv9c"
Sep 09 05:50:08.072140 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.072118    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz58p\" (UniqueName: \"kubernetes.io/projected/6cd6c398-b2b7-4c95-8899-18f779265279-kube-api-access-rz58p\") pod \"service-ca-operator-859f94c4b-8hmqm\" (UID: \"6cd6c398-b2b7-4c95-8899-18f779265279\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm"
Sep 09 05:50:08.072425 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.072407    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d44x4\" (UniqueName: \"kubernetes.io/projected/be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8-kube-api-access-d44x4\") pod \"ingress-canary-9gc29\" (UID: \"be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8\") " pod="openshift-ingress-canary/ingress-canary-9gc29"
Sep 09 05:50:08.083719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.083704    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp"
Sep 09 05:50:08.099966 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.099943    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm"
Sep 09 05:50:08.117809 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.117782    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:08.139486 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.139271    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-4wv9c"
Sep 09 05:50:08.166920 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.166314    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f3f129d-c477-4771-a0cd-4c9e710bb832-tmp-dir\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:08.166920 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.166387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:08.166920 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.166481    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3f129d-c477-4771-a0cd-4c9e710bb832-config-volume\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:08.166920 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.166527    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnzw9\" (UniqueName: \"kubernetes.io/projected/9f3f129d-c477-4771-a0cd-4c9e710bb832-kube-api-access-dnzw9\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:08.166920 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.166626    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 05:50:08.166920 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.166687    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls podName:9f3f129d-c477-4771-a0cd-4c9e710bb832 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:08.666668835 +0000 UTC m=+35.882998701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls") pod "dns-default-pvjjm" (UID: "9f3f129d-c477-4771-a0cd-4c9e710bb832") : secret "dns-default-metrics-tls" not found
Sep 09 05:50:08.168079 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.167485    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f3f129d-c477-4771-a0cd-4c9e710bb832-tmp-dir\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:08.168375 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.168037    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3f129d-c477-4771-a0cd-4c9e710bb832-config-volume\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:08.169981 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.169961    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz"
Sep 09 05:50:08.183218 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.183192    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnzw9\" (UniqueName: \"kubernetes.io/projected/9f3f129d-c477-4771-a0cd-4c9e710bb832-kube-api-access-dnzw9\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:08.190155 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.190133    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9gc29"
Sep 09 05:50:08.296429 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.296320    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:50:08.299989 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.299965    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hh975\""
Sep 09 05:50:08.368442 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.368412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:08.368615 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.368592    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 05:50:08.368615 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.368614    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64cdf68b7d-mdwmv: secret "image-registry-tls" not found
Sep 09 05:50:08.368729 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.368682    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls podName:48cf9622-0adf-4129-955f-3e82431ab4ab nodeName:}" failed. No retries permitted until 2026-09-09 05:50:09.368662463 +0000 UTC m=+36.584992341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls") pod "image-registry-64cdf68b7d-mdwmv" (UID: "48cf9622-0adf-4129-955f-3e82431ab4ab") : secret "image-registry-tls" not found
Sep 09 05:50:08.379363 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.379337    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qb2nd"]
Sep 09 05:50:08.382375 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:08.382345    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5d31e4_20fc_48c9_935d_087f65b77eca.slice/crio-4d5426dfb59b8b36bb9d129d875ceaf829729db29989b230ae6cdc23c0b0e96c WatchSource:0}: Error finding container 4d5426dfb59b8b36bb9d129d875ceaf829729db29989b230ae6cdc23c0b0e96c: Status 404 returned error can't find the container with id 4d5426dfb59b8b36bb9d129d875ceaf829729db29989b230ae6cdc23c0b0e96c
Sep 09 05:50:08.387327 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.387305    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp"]
Sep 09 05:50:08.389839 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:08.389819    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ce177a_74d8_4240_8eb1_bf8c703e59d6.slice/crio-97b09eec9209e4526324d1a8020caa7b3efafaeb6e94d06ed75b083f28bf3cd0 WatchSource:0}: Error finding container 97b09eec9209e4526324d1a8020caa7b3efafaeb6e94d06ed75b083f28bf3cd0: Status 404 returned error can't find the container with id 97b09eec9209e4526324d1a8020caa7b3efafaeb6e94d06ed75b083f28bf3cd0
Sep 09 05:50:08.422193 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.422170    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm"]
Sep 09 05:50:08.424627 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:08.424604    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd6c398_b2b7_4c95_8899_18f779265279.slice/crio-64cd0c977e495b9b59f7657e106d9b2d245e2a1bf9e8b4449e320f64b6353eb6 WatchSource:0}: Error finding container 64cd0c977e495b9b59f7657e106d9b2d245e2a1bf9e8b4449e320f64b6353eb6: Status 404 returned error can't find the container with id 64cd0c977e495b9b59f7657e106d9b2d245e2a1bf9e8b4449e320f64b6353eb6
Sep 09 05:50:08.445718 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.445674    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qb2nd" event={"ID":"be5d31e4-20fc-48c9-935d-087f65b77eca","Type":"ContainerStarted","Data":"4d5426dfb59b8b36bb9d129d875ceaf829729db29989b230ae6cdc23c0b0e96c"}
Sep 09 05:50:08.446821 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.446803    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-5mbcw"]
Sep 09 05:50:08.449003 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:08.448984    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207d756d_5ccd_456e_bd3f_3ede12b60f78.slice/crio-6b2b7b9b7e84d73089d5649fcb128cda7ff67f524dd7b9c5be33591419cb8e5e WatchSource:0}: Error finding container 6b2b7b9b7e84d73089d5649fcb128cda7ff67f524dd7b9c5be33591419cb8e5e: Status 404 returned error can't find the container with id 6b2b7b9b7e84d73089d5649fcb128cda7ff67f524dd7b9c5be33591419cb8e5e
Sep 09 05:50:08.449314 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.449296    2564 generic.go:356] "Generic (PLEG): container finished" podID="9d280b22-a809-4323-8a18-2049d0394312" containerID="81ad99dda2c3c77754b488a524de7ef147858b88eb51380e5bded81c38ae29c6" exitCode=0
Sep 09 05:50:08.449407 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.449358    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz2h" event={"ID":"9d280b22-a809-4323-8a18-2049d0394312","Type":"ContainerDied","Data":"81ad99dda2c3c77754b488a524de7ef147858b88eb51380e5bded81c38ae29c6"}
Sep 09 05:50:08.450618 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.450520    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm" event={"ID":"6cd6c398-b2b7-4c95-8899-18f779265279","Type":"ContainerStarted","Data":"64cd0c977e495b9b59f7657e106d9b2d245e2a1bf9e8b4449e320f64b6353eb6"}
Sep 09 05:50:08.451709 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.451692    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp" event={"ID":"c2ce177a-74d8-4240-8eb1-bf8c703e59d6","Type":"ContainerStarted","Data":"97b09eec9209e4526324d1a8020caa7b3efafaeb6e94d06ed75b083f28bf3cd0"}
Sep 09 05:50:08.468812 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.468787    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jbrn2\" (UID: \"cf0f9476-232d-4dfb-8b60-175fba9847db\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"
Sep 09 05:50:08.468905 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.468866    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:08.468978 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.468916    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:08.468978 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.468928    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 05:50:08.468978 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.468961    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7lfm4\" (UID: \"f163cf03-e312-4296-8fd6-cc95260b51b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4"
Sep 09 05:50:08.469118 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.469001    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 05:50:08.469118 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.469007    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls podName:cf0f9476-232d-4dfb-8b60-175fba9847db nodeName:}" failed. No retries permitted until 2026-09-09 05:50:09.468986598 +0000 UTC m=+36.685316480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jbrn2" (UID: "cf0f9476-232d-4dfb-8b60-175fba9847db") : secret "samples-operator-tls" not found
Sep 09 05:50:08.469118 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.469055    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 05:50:08.469118 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.469093    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-prgks\" (UID: \"f04421dd-ed42-46c3-997e-9bf3d4359d7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:08.469267 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.469143    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert podName:f163cf03-e312-4296-8fd6-cc95260b51b3 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:09.469097499 +0000 UTC m=+36.685427363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-7lfm4" (UID: "f163cf03-e312-4296-8fd6-cc95260b51b3") : secret "networking-console-plugin-cert" not found
Sep 09 05:50:08.469267 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.469155    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 05:50:08.469267 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.469185    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs podName:08804660-c36f-432a-80c1-6baba82c301f nodeName:}" failed. No retries permitted until 2026-09-09 05:50:09.469157766 +0000 UTC m=+36.685487638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs") pod "router-default-78695b457d-lfsxh" (UID: "08804660-c36f-432a-80c1-6baba82c301f") : secret "router-metrics-certs-default" not found
Sep 09 05:50:08.469267 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.469232    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle podName:08804660-c36f-432a-80c1-6baba82c301f nodeName:}" failed. No retries permitted until 2026-09-09 05:50:09.469207103 +0000 UTC m=+36.685536999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle") pod "router-default-78695b457d-lfsxh" (UID: "08804660-c36f-432a-80c1-6baba82c301f") : configmap references non-existent config key: service-ca.crt
Sep 09 05:50:08.469391 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.469312    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls podName:f04421dd-ed42-46c3-997e-9bf3d4359d7a nodeName:}" failed. No retries permitted until 2026-09-09 05:50:09.469301625 +0000 UTC m=+36.685631581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-prgks" (UID: "f04421dd-ed42-46c3-997e-9bf3d4359d7a") : secret "cluster-monitoring-operator-tls" not found
Sep 09 05:50:08.481577 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.481550    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-4wv9c"]
Sep 09 05:50:08.484921 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:08.484895    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7a5caf_c270_423f_919e_e7cb1a10a8ff.slice/crio-21386359a2c667463a94a08dcd5158e64d2b1216d603677e64ac87df41d1c2a4 WatchSource:0}: Error finding container 21386359a2c667463a94a08dcd5158e64d2b1216d603677e64ac87df41d1c2a4: Status 404 returned error can't find the container with id 21386359a2c667463a94a08dcd5158e64d2b1216d603677e64ac87df41d1c2a4
Sep 09 05:50:08.511591 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.511563    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-cflnz"]
Sep 09 05:50:08.515197 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:08.515171    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b76aac_f159_4940_8a85_64a227c18b05.slice/crio-40cf44d7ec0bfa26f2dbaf1ef27cc700f54a28bbb365ca61c4ea2658ff6ea771 WatchSource:0}: Error finding container 40cf44d7ec0bfa26f2dbaf1ef27cc700f54a28bbb365ca61c4ea2658ff6ea771: Status 404 returned error can't find the container with id 40cf44d7ec0bfa26f2dbaf1ef27cc700f54a28bbb365ca61c4ea2658ff6ea771
Sep 09 05:50:08.530019 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.529994    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9gc29"]
Sep 09 05:50:08.533476 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:08.533448    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe276cf4_4edd_4f8e_a2fa_1c2e9deed8f8.slice/crio-826060b3b943a78ea3cf24fd90f6a7547ab59c6b2e02845255819a276f6a8966 WatchSource:0}: Error finding container 826060b3b943a78ea3cf24fd90f6a7547ab59c6b2e02845255819a276f6a8966: Status 404 returned error can't find the container with id 826060b3b943a78ea3cf24fd90f6a7547ab59c6b2e02845255819a276f6a8966
Sep 09 05:50:08.672306 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:08.672275    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:08.672453 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.672422    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 05:50:08.672551 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:08.672508    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls podName:9f3f129d-c477-4771-a0cd-4c9e710bb832 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:09.672472989 +0000 UTC m=+36.888802874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls") pod "dns-default-pvjjm" (UID: "9f3f129d-c477-4771-a0cd-4c9e710bb832") : secret "dns-default-metrics-tls" not found
Sep 09 05:50:09.299283 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.299250    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:50:09.299731 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.299712    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:50:09.303718 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.302451    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 05:50:09.303718 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.302694    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 05:50:09.303718 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.302902    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnkgt\""
Sep 09 05:50:09.380480 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.379874    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:09.380480 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:09.380061    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 05:50:09.380480 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:09.380077    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64cdf68b7d-mdwmv: secret "image-registry-tls" not found
Sep 09 05:50:09.380480 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:09.380134    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls podName:48cf9622-0adf-4129-955f-3e82431ab4ab nodeName:}" failed. No retries permitted until 2026-09-09 05:50:11.380115432 +0000 UTC m=+38.596445299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls") pod "image-registry-64cdf68b7d-mdwmv" (UID: "48cf9622-0adf-4129-955f-3e82431ab4ab") : secret "image-registry-tls" not found
Sep 09 05:50:09.460634 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.460556    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9gc29" event={"ID":"be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8","Type":"ContainerStarted","Data":"826060b3b943a78ea3cf24fd90f6a7547ab59c6b2e02845255819a276f6a8966"}
Sep 09 05:50:09.462646 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.462575    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-4wv9c" event={"ID":"ef7a5caf-c270-423f-919e-e7cb1a10a8ff","Type":"ContainerStarted","Data":"21386359a2c667463a94a08dcd5158e64d2b1216d603677e64ac87df41d1c2a4"}
Sep 09 05:50:09.466517 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.466384    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-5mbcw" event={"ID":"207d756d-5ccd-456e-bd3f-3ede12b60f78","Type":"ContainerStarted","Data":"6b2b7b9b7e84d73089d5649fcb128cda7ff67f524dd7b9c5be33591419cb8e5e"}
Sep 09 05:50:09.469219 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.469146    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz" event={"ID":"d1b76aac-f159-4940-8a85-64a227c18b05","Type":"ContainerStarted","Data":"40cf44d7ec0bfa26f2dbaf1ef27cc700f54a28bbb365ca61c4ea2658ff6ea771"}
Sep 09 05:50:09.479068 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.479038    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz2h" event={"ID":"9d280b22-a809-4323-8a18-2049d0394312","Type":"ContainerStarted","Data":"7b5accde5d354ff55ed568265efc570cf1984a7c7b658b75cb4e59f4e42930f0"}
Sep 09 05:50:09.480539 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.480514    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jbrn2\" (UID: \"cf0f9476-232d-4dfb-8b60-175fba9847db\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"
Sep 09 05:50:09.480655 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.480607    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:09.480717 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.480658    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:09.480717 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.480706    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7lfm4\" (UID: \"f163cf03-e312-4296-8fd6-cc95260b51b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4"
Sep 09 05:50:09.480812 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.480776    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-prgks\" (UID: \"f04421dd-ed42-46c3-997e-9bf3d4359d7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:09.480949 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:09.480921    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 05:50:09.481009 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:09.480980    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls podName:f04421dd-ed42-46c3-997e-9bf3d4359d7a nodeName:}" failed. No retries permitted until 2026-09-09 05:50:11.480962777 +0000 UTC m=+38.697292656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-prgks" (UID: "f04421dd-ed42-46c3-997e-9bf3d4359d7a") : secret "cluster-monitoring-operator-tls" not found
Sep 09 05:50:09.481403 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:09.481387    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 05:50:09.481504 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:09.481438    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls podName:cf0f9476-232d-4dfb-8b60-175fba9847db nodeName:}" failed. No retries permitted until 2026-09-09 05:50:11.481423876 +0000 UTC m=+38.697753752 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jbrn2" (UID: "cf0f9476-232d-4dfb-8b60-175fba9847db") : secret "samples-operator-tls" not found
Sep 09 05:50:09.481576 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:09.481516    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 05:50:09.481576 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:09.481549    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs podName:08804660-c36f-432a-80c1-6baba82c301f nodeName:}" failed. No retries permitted until 2026-09-09 05:50:11.481537758 +0000 UTC m=+38.697867629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs") pod "router-default-78695b457d-lfsxh" (UID: "08804660-c36f-432a-80c1-6baba82c301f") : secret "router-metrics-certs-default" not found
Sep 09 05:50:09.481691 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:09.481613    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle podName:08804660-c36f-432a-80c1-6baba82c301f nodeName:}" failed. No retries permitted until 2026-09-09 05:50:11.481603269 +0000 UTC m=+38.697933136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle") pod "router-default-78695b457d-lfsxh" (UID: "08804660-c36f-432a-80c1-6baba82c301f") : configmap references non-existent config key: service-ca.crt
Sep 09 05:50:09.481691 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:09.481682    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 05:50:09.481801 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:09.481714    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert podName:f163cf03-e312-4296-8fd6-cc95260b51b3 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:11.48170364 +0000 UTC m=+38.698033505 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-7lfm4" (UID: "f163cf03-e312-4296-8fd6-cc95260b51b3") : secret "networking-console-plugin-cert" not found
Sep 09 05:50:09.500156 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.500103    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wrz2h" podStartSLOduration=7.449192632 podStartE2EDuration="36.5000822s" totalImagesPullingTime="29.050889568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:09.499023108 +0000 UTC m=+36.715353016" watchObservedRunningTime="2026-09-09 05:50:09.5000822 +0000 UTC m=+36.716412085"
Sep 09 05:50:09.682368 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:09.682310    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:09.682673 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:09.682656    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 05:50:09.682765 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:09.682723    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls podName:9f3f129d-c477-4771-a0cd-4c9e710bb832 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:11.682704351 +0000 UTC m=+38.899034216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls") pod "dns-default-pvjjm" (UID: "9f3f129d-c477-4771-a0cd-4c9e710bb832") : secret "dns-default-metrics-tls" not found
Sep 09 05:50:11.399165 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:11.398943    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:11.399618 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:11.399108    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 05:50:11.399618 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:11.399249    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64cdf68b7d-mdwmv: secret "image-registry-tls" not found
Sep 09 05:50:11.399618 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:11.399318    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls podName:48cf9622-0adf-4129-955f-3e82431ab4ab nodeName:}" failed. No retries permitted until 2026-09-09 05:50:15.399297767 +0000 UTC m=+42.615627635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls") pod "image-registry-64cdf68b7d-mdwmv" (UID: "48cf9622-0adf-4129-955f-3e82431ab4ab") : secret "image-registry-tls" not found
Sep 09 05:50:11.499866 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:11.499841    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:11.500043 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:11.499895    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:11.500043 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:11.499936    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7lfm4\" (UID: \"f163cf03-e312-4296-8fd6-cc95260b51b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4"
Sep 09 05:50:11.500043 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:11.499984    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-prgks\" (UID: \"f04421dd-ed42-46c3-997e-9bf3d4359d7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:11.500043 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:11.499994    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 05:50:11.500212 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:11.500059    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 05:50:11.500212 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:11.500061    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 05:50:11.500212 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:11.500065    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs podName:08804660-c36f-432a-80c1-6baba82c301f nodeName:}" failed. No retries permitted until 2026-09-09 05:50:15.50004578 +0000 UTC m=+42.716375662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs") pod "router-default-78695b457d-lfsxh" (UID: "08804660-c36f-432a-80c1-6baba82c301f") : secret "router-metrics-certs-default" not found
Sep 09 05:50:11.500212 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:11.500162    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jbrn2\" (UID: \"cf0f9476-232d-4dfb-8b60-175fba9847db\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"
Sep 09 05:50:11.500421 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:11.500212    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle podName:08804660-c36f-432a-80c1-6baba82c301f nodeName:}" failed. No retries permitted until 2026-09-09 05:50:15.50018744 +0000 UTC m=+42.716517317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle") pod "router-default-78695b457d-lfsxh" (UID: "08804660-c36f-432a-80c1-6baba82c301f") : configmap references non-existent config key: service-ca.crt
Sep 09 05:50:11.500421 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:11.500233    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 05:50:11.500421 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:11.500266    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls podName:f04421dd-ed42-46c3-997e-9bf3d4359d7a nodeName:}" failed. No retries permitted until 2026-09-09 05:50:15.50023613 +0000 UTC m=+42.716565994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-prgks" (UID: "f04421dd-ed42-46c3-997e-9bf3d4359d7a") : secret "cluster-monitoring-operator-tls" not found
Sep 09 05:50:11.500421 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:11.500288    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls podName:cf0f9476-232d-4dfb-8b60-175fba9847db nodeName:}" failed. No retries permitted until 2026-09-09 05:50:15.500278604 +0000 UTC m=+42.716608466 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jbrn2" (UID: "cf0f9476-232d-4dfb-8b60-175fba9847db") : secret "samples-operator-tls" not found
Sep 09 05:50:11.500421 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:11.500307    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert podName:f163cf03-e312-4296-8fd6-cc95260b51b3 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:15.500298843 +0000 UTC m=+42.716628744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-7lfm4" (UID: "f163cf03-e312-4296-8fd6-cc95260b51b3") : secret "networking-console-plugin-cert" not found
Sep 09 05:50:11.702014 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:11.701942    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:11.702154 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:11.702082    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 05:50:11.702204 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:11.702157    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls podName:9f3f129d-c477-4771-a0cd-4c9e710bb832 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:15.70213628 +0000 UTC m=+42.918466164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls") pod "dns-default-pvjjm" (UID: "9f3f129d-c477-4771-a0cd-4c9e710bb832") : secret "dns-default-metrics-tls" not found
Sep 09 05:50:15.435470 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:15.435423    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:15.435943 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:15.435605    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 05:50:15.435943 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:15.435619    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64cdf68b7d-mdwmv: secret "image-registry-tls" not found
Sep 09 05:50:15.435943 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:15.435681    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls podName:48cf9622-0adf-4129-955f-3e82431ab4ab nodeName:}" failed. No retries permitted until 2026-09-09 05:50:23.435664558 +0000 UTC m=+50.651994423 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls") pod "image-registry-64cdf68b7d-mdwmv" (UID: "48cf9622-0adf-4129-955f-3e82431ab4ab") : secret "image-registry-tls" not found
Sep 09 05:50:15.536067 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:15.536025    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:15.536251 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:15.536100    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:15.536251 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:15.536148    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7lfm4\" (UID: \"f163cf03-e312-4296-8fd6-cc95260b51b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4"
Sep 09 05:50:15.536251 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:15.536187    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 05:50:15.536427 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:15.536256    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs podName:08804660-c36f-432a-80c1-6baba82c301f nodeName:}" failed. No retries permitted until 2026-09-09 05:50:23.536236135 +0000 UTC m=+50.752566016 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs") pod "router-default-78695b457d-lfsxh" (UID: "08804660-c36f-432a-80c1-6baba82c301f") : secret "router-metrics-certs-default" not found
Sep 09 05:50:15.536427 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:15.536263    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 05:50:15.536427 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:15.536280    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle podName:08804660-c36f-432a-80c1-6baba82c301f nodeName:}" failed. No retries permitted until 2026-09-09 05:50:23.536263202 +0000 UTC m=+50.752593066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle") pod "router-default-78695b457d-lfsxh" (UID: "08804660-c36f-432a-80c1-6baba82c301f") : configmap references non-existent config key: service-ca.crt
Sep 09 05:50:15.536427 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:15.536332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-prgks\" (UID: \"f04421dd-ed42-46c3-997e-9bf3d4359d7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:15.536427 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:15.536341    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert podName:f163cf03-e312-4296-8fd6-cc95260b51b3 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:23.536322719 +0000 UTC m=+50.752652588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-7lfm4" (UID: "f163cf03-e312-4296-8fd6-cc95260b51b3") : secret "networking-console-plugin-cert" not found
Sep 09 05:50:15.536427 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:15.536398    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 05:50:15.536742 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:15.536426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jbrn2\" (UID: \"cf0f9476-232d-4dfb-8b60-175fba9847db\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"
Sep 09 05:50:15.536742 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:15.536438    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls podName:f04421dd-ed42-46c3-997e-9bf3d4359d7a nodeName:}" failed. No retries permitted until 2026-09-09 05:50:23.536426394 +0000 UTC m=+50.752756256 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-prgks" (UID: "f04421dd-ed42-46c3-997e-9bf3d4359d7a") : secret "cluster-monitoring-operator-tls" not found
Sep 09 05:50:15.536742 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:15.536481    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 05:50:15.536742 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:15.536535    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls podName:cf0f9476-232d-4dfb-8b60-175fba9847db nodeName:}" failed. No retries permitted until 2026-09-09 05:50:23.536524162 +0000 UTC m=+50.752854029 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jbrn2" (UID: "cf0f9476-232d-4dfb-8b60-175fba9847db") : secret "samples-operator-tls" not found
Sep 09 05:50:15.737607 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:15.737483    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:15.737607 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:15.737598    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 05:50:15.737836 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:15.737667    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls podName:9f3f129d-c477-4771-a0cd-4c9e710bb832 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:23.737650253 +0000 UTC m=+50.953980114 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls") pod "dns-default-pvjjm" (UID: "9f3f129d-c477-4771-a0cd-4c9e710bb832") : secret "dns-default-metrics-tls" not found
Sep 09 05:50:16.502079 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:16.501983    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz" event={"ID":"d1b76aac-f159-4940-8a85-64a227c18b05","Type":"ContainerStarted","Data":"59eb5f42f4334dafadefe210edf8b975048ff08744534cf1dff8a7484feb0bb8"}
Sep 09 05:50:16.504321 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:16.504295    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm" event={"ID":"6cd6c398-b2b7-4c95-8899-18f779265279","Type":"ContainerStarted","Data":"0530d4c5697a48ea53dc89f625a0b3291295d251310fad0ec18f100581ec13fe"}
Sep 09 05:50:16.506139 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:16.506115    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qb2nd" event={"ID":"be5d31e4-20fc-48c9-935d-087f65b77eca","Type":"ContainerStarted","Data":"3d7bb6f7bb42be2a0122729a2ded4ecce42546bf48aa7f8e05fc0a0cb87291bc"}
Sep 09 05:50:16.535366 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:16.535309    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm" podStartSLOduration=9.656749625 podStartE2EDuration="17.535286345s" totalImagesPullingTime="7.87853672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:16.533817925 +0000 UTC m=+43.750147809" watchObservedRunningTime="2026-09-09 05:50:16.535286345 +0000 UTC m=+43.751616226"
Sep 09 05:50:16.559682 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:16.559630    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qb2nd" podStartSLOduration=9.783854932 podStartE2EDuration="17.559608997s" totalImagesPullingTime="7.775754065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:16.558163052 +0000 UTC m=+43.774492946" watchObservedRunningTime="2026-09-09 05:50:16.559608997 +0000 UTC m=+43.775938883"
Sep 09 05:50:17.410081 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.410048    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hc2v2"]
Sep 09 05:50:17.444600 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.444566    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hc2v2"]
Sep 09 05:50:17.444750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.444636    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:17.447076 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.447054    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 05:50:17.447232 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.447211    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xqjvr\""
Sep 09 05:50:17.447351 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.447209    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 05:50:17.456544 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.456520    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf5v8\" (UniqueName: \"kubernetes.io/projected/23b784c5-0b83-493f-917e-559bc6a26b72-kube-api-access-tf5v8\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:17.456654 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.456550    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23b784c5-0b83-493f-917e-559bc6a26b72-data-volume\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:17.456654 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.456595    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b784c5-0b83-493f-917e-559bc6a26b72-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:17.456763 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.456698    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:17.456830 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.456811    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23b784c5-0b83-493f-917e-559bc6a26b72-crio-socket\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:17.511508 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.511454    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9gc29" event={"ID":"be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8","Type":"ContainerStarted","Data":"2ac95e2379493bae7a6222f09080d00a6193f17fc618d02ac9cb44d798368d4a"}
Sep 09 05:50:17.513392 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.513353    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-4wv9c" event={"ID":"ef7a5caf-c270-423f-919e-e7cb1a10a8ff","Type":"ContainerStarted","Data":"ffdef434f0026053391ecddd318d9e324fd565c9bc84f9bf93f154bba8c8bb69"}
Sep 09 05:50:17.515000 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.514977    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/0.log"
Sep 09 05:50:17.515118 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.515014    2564 generic.go:356] "Generic (PLEG): container finished" podID="207d756d-5ccd-456e-bd3f-3ede12b60f78" containerID="3d08148c65045dabe15adcf151574178e9da06b8eb0528c7e5b37059a9ac7709" exitCode=255
Sep 09 05:50:17.515118 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.515041    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-5mbcw" event={"ID":"207d756d-5ccd-456e-bd3f-3ede12b60f78","Type":"ContainerDied","Data":"3d08148c65045dabe15adcf151574178e9da06b8eb0528c7e5b37059a9ac7709"}
Sep 09 05:50:17.515607 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.515590    2564 scope.go:117] "RemoveContainer" containerID="3d08148c65045dabe15adcf151574178e9da06b8eb0528c7e5b37059a9ac7709"
Sep 09 05:50:17.517206 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.516874    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp" event={"ID":"c2ce177a-74d8-4240-8eb1-bf8c703e59d6","Type":"ContainerStarted","Data":"d98f2a9cd9cb73fc8b5fd73c3cbeac9fc836889a5405f6489e93331f0d82a168"}
Sep 09 05:50:17.528823 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.528780    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9gc29" podStartSLOduration=2.7590508099999997 podStartE2EDuration="10.528767871s" totalImagesPullingTime="7.769717061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:17.527559096 +0000 UTC m=+44.743888990" watchObservedRunningTime="2026-09-09 05:50:17.528767871 +0000 UTC m=+44.745097754"
Sep 09 05:50:17.546595 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.546517    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz" podStartSLOduration=10.75875558 podStartE2EDuration="18.546484348s" totalImagesPullingTime="7.787728768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:17.545347591 +0000 UTC m=+44.761677477" watchObservedRunningTime="2026-09-09 05:50:17.546484348 +0000 UTC m=+44.762814230"
Sep 09 05:50:17.558257 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.558229    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf5v8\" (UniqueName: \"kubernetes.io/projected/23b784c5-0b83-493f-917e-559bc6a26b72-kube-api-access-tf5v8\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:17.558373 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.558292    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23b784c5-0b83-493f-917e-559bc6a26b72-data-volume\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:17.558441 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.558373    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b784c5-0b83-493f-917e-559bc6a26b72-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:17.558512 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.558468    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:17.558589 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.558571    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23b784c5-0b83-493f-917e-559bc6a26b72-crio-socket\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:17.558915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.558891    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23b784c5-0b83-493f-917e-559bc6a26b72-data-volume\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:17.559214 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:17.559197    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 05:50:17.559288 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:17.559258    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23b784c5-0b83-493f-917e-559bc6a26b72-kube-rbac-proxy-cm podName:23b784c5-0b83-493f-917e-559bc6a26b72 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:18.059240127 +0000 UTC m=+45.275570005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23b784c5-0b83-493f-917e-559bc6a26b72-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hc2v2" (UID: "23b784c5-0b83-493f-917e-559bc6a26b72") : configmap "kube-rbac-proxy" not found
Sep 09 05:50:17.559656 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:17.559637    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:17.559731 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:17.559688    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls podName:23b784c5-0b83-493f-917e-559bc6a26b72 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:18.059673997 +0000 UTC m=+45.276003864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hc2v2" (UID: "23b784c5-0b83-493f-917e-559bc6a26b72") : secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:17.559836 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.559817    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23b784c5-0b83-493f-917e-559bc6a26b72-crio-socket\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:17.566356 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.566295    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp" podStartSLOduration=10.652365083 podStartE2EDuration="18.566277488s" totalImagesPullingTime="7.913912405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:17.564138744 +0000 UTC m=+44.780468633" watchObservedRunningTime="2026-09-09 05:50:17.566277488 +0000 UTC m=+44.782607373"
Sep 09 05:50:17.572834 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.572773    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf5v8\" (UniqueName: \"kubernetes.io/projected/23b784c5-0b83-493f-917e-559bc6a26b72-kube-api-access-tf5v8\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:17.580733 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:17.580685    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-4wv9c" podStartSLOduration=10.752945155 podStartE2EDuration="18.580667667s" totalImagesPullingTime="7.827722512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:17.580527586 +0000 UTC m=+44.796857463" watchObservedRunningTime="2026-09-09 05:50:17.580667667 +0000 UTC m=+44.796997552"
Sep 09 05:50:18.062557 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:18.062452    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b784c5-0b83-493f-917e-559bc6a26b72-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:18.062557 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:18.062542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:18.062782 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:18.062575    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 05:50:18.062782 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:18.062643    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23b784c5-0b83-493f-917e-559bc6a26b72-kube-rbac-proxy-cm podName:23b784c5-0b83-493f-917e-559bc6a26b72 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:19.062626064 +0000 UTC m=+46.278955946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23b784c5-0b83-493f-917e-559bc6a26b72-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hc2v2" (UID: "23b784c5-0b83-493f-917e-559bc6a26b72") : configmap "kube-rbac-proxy" not found
Sep 09 05:50:18.062782 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:18.062706    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:18.062782 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:18.062772    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls podName:23b784c5-0b83-493f-917e-559bc6a26b72 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:19.062756565 +0000 UTC m=+46.279086433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hc2v2" (UID: "23b784c5-0b83-493f-917e-559bc6a26b72") : secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:18.118903 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:18.118870    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:18.119059 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:18.118918    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:18.522146 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:18.522122    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 05:50:18.522544 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:18.522521    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/0.log"
Sep 09 05:50:18.522608 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:18.522552    2564 generic.go:356] "Generic (PLEG): container finished" podID="207d756d-5ccd-456e-bd3f-3ede12b60f78" containerID="64b79b471744a4fd1c08e55701eaf818222cb57ce968076446adb4b7f2b9e640" exitCode=255
Sep 09 05:50:18.522656 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:18.522632    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-5mbcw" event={"ID":"207d756d-5ccd-456e-bd3f-3ede12b60f78","Type":"ContainerDied","Data":"64b79b471744a4fd1c08e55701eaf818222cb57ce968076446adb4b7f2b9e640"}
Sep 09 05:50:18.522698 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:18.522676    2564 scope.go:117] "RemoveContainer" containerID="3d08148c65045dabe15adcf151574178e9da06b8eb0528c7e5b37059a9ac7709"
Sep 09 05:50:18.522909 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:18.522888    2564 scope.go:117] "RemoveContainer" containerID="64b79b471744a4fd1c08e55701eaf818222cb57ce968076446adb4b7f2b9e640"
Sep 09 05:50:18.523130 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:18.523098    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-5mbcw_openshift-console-operator(207d756d-5ccd-456e-bd3f-3ede12b60f78)\"" pod="openshift-console-operator/console-operator-85b454874b-5mbcw" podUID="207d756d-5ccd-456e-bd3f-3ede12b60f78"
Sep 09 05:50:19.071796 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:19.071756    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b784c5-0b83-493f-917e-559bc6a26b72-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:19.071971 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:19.071821    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:19.071971 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:19.071877    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 05:50:19.071971 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:19.071949    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23b784c5-0b83-493f-917e-559bc6a26b72-kube-rbac-proxy-cm podName:23b784c5-0b83-493f-917e-559bc6a26b72 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:21.071929364 +0000 UTC m=+48.288259245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23b784c5-0b83-493f-917e-559bc6a26b72-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hc2v2" (UID: "23b784c5-0b83-493f-917e-559bc6a26b72") : configmap "kube-rbac-proxy" not found
Sep 09 05:50:19.071971 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:19.071958    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:19.072126 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:19.071993    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls podName:23b784c5-0b83-493f-917e-559bc6a26b72 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:21.071981584 +0000 UTC m=+48.288311460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hc2v2" (UID: "23b784c5-0b83-493f-917e-559bc6a26b72") : secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:19.389272 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:19.389206    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h8v7x_40030ec2-6b02-4d13-b794-aa551a09290e/dns-node-resolver/0.log"
Sep 09 05:50:19.527396 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:19.527369    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 05:50:19.527882 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:19.527791    2564 scope.go:117] "RemoveContainer" containerID="64b79b471744a4fd1c08e55701eaf818222cb57ce968076446adb4b7f2b9e640"
Sep 09 05:50:19.528001 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:19.527980    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-5mbcw_openshift-console-operator(207d756d-5ccd-456e-bd3f-3ede12b60f78)\"" pod="openshift-console-operator/console-operator-85b454874b-5mbcw" podUID="207d756d-5ccd-456e-bd3f-3ede12b60f78"
Sep 09 05:50:20.583462 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:20.583434    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dmx8r_f4234e30-8f0b-47d1-a67d-36deddc3a89c/node-ca/0.log"
Sep 09 05:50:20.983546 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:20.983442    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9gc29_be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8/serve-healthcheck-canary/0.log"
Sep 09 05:50:21.086764 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:21.086737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b784c5-0b83-493f-917e-559bc6a26b72-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:21.086920 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:21.086788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:21.086920 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:21.086860    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 05:50:21.086920 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:21.086884    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:21.087019 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:21.086925    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23b784c5-0b83-493f-917e-559bc6a26b72-kube-rbac-proxy-cm podName:23b784c5-0b83-493f-917e-559bc6a26b72 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:25.086906935 +0000 UTC m=+52.303236797 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23b784c5-0b83-493f-917e-559bc6a26b72-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hc2v2" (UID: "23b784c5-0b83-493f-917e-559bc6a26b72") : configmap "kube-rbac-proxy" not found
Sep 09 05:50:21.087019 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:21.086950    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls podName:23b784c5-0b83-493f-917e-559bc6a26b72 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:25.086942098 +0000 UTC m=+52.303271967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hc2v2" (UID: "23b784c5-0b83-493f-917e-559bc6a26b72") : secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:23.509554 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:23.509522    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:23.509928 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:23.509625    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 05:50:23.509928 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:23.509645    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64cdf68b7d-mdwmv: secret "image-registry-tls" not found
Sep 09 05:50:23.509928 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:23.509703    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls podName:48cf9622-0adf-4129-955f-3e82431ab4ab nodeName:}" failed. No retries permitted until 2026-09-09 05:50:39.509683237 +0000 UTC m=+66.726013099 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls") pod "image-registry-64cdf68b7d-mdwmv" (UID: "48cf9622-0adf-4129-955f-3e82431ab4ab") : secret "image-registry-tls" not found
Sep 09 05:50:23.610943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:23.610911    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-prgks\" (UID: \"f04421dd-ed42-46c3-997e-9bf3d4359d7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:23.611092 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:23.610968    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jbrn2\" (UID: \"cf0f9476-232d-4dfb-8b60-175fba9847db\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"
Sep 09 05:50:23.611092 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:23.611045    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 05:50:23.611092 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:23.611048    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 05:50:23.611092 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:23.611087    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls podName:cf0f9476-232d-4dfb-8b60-175fba9847db nodeName:}" failed. No retries permitted until 2026-09-09 05:50:39.611075793 +0000 UTC m=+66.827405655 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jbrn2" (UID: "cf0f9476-232d-4dfb-8b60-175fba9847db") : secret "samples-operator-tls" not found
Sep 09 05:50:23.611242 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:23.611107    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls podName:f04421dd-ed42-46c3-997e-9bf3d4359d7a nodeName:}" failed. No retries permitted until 2026-09-09 05:50:39.611092797 +0000 UTC m=+66.827422659 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-prgks" (UID: "f04421dd-ed42-46c3-997e-9bf3d4359d7a") : secret "cluster-monitoring-operator-tls" not found
Sep 09 05:50:23.611242 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:23.611170    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:23.611242 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:23.611206    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:23.611359 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:23.611244    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7lfm4\" (UID: \"f163cf03-e312-4296-8fd6-cc95260b51b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4"
Sep 09 05:50:23.611359 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:23.611297    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 05:50:23.611359 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:23.611322    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle podName:08804660-c36f-432a-80c1-6baba82c301f nodeName:}" failed. No retries permitted until 2026-09-09 05:50:39.611310521 +0000 UTC m=+66.827640383 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle") pod "router-default-78695b457d-lfsxh" (UID: "08804660-c36f-432a-80c1-6baba82c301f") : configmap references non-existent config key: service-ca.crt
Sep 09 05:50:23.611448 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:23.611365    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 05:50:23.611448 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:23.611384    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs podName:08804660-c36f-432a-80c1-6baba82c301f nodeName:}" failed. No retries permitted until 2026-09-09 05:50:39.611368782 +0000 UTC m=+66.827698658 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs") pod "router-default-78695b457d-lfsxh" (UID: "08804660-c36f-432a-80c1-6baba82c301f") : secret "router-metrics-certs-default" not found
Sep 09 05:50:23.611448 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:23.611405    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert podName:f163cf03-e312-4296-8fd6-cc95260b51b3 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:39.611394464 +0000 UTC m=+66.827724340 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-7lfm4" (UID: "f163cf03-e312-4296-8fd6-cc95260b51b3") : secret "networking-console-plugin-cert" not found
Sep 09 05:50:23.813027 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:23.812950    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:23.813161 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:23.813100    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 05:50:23.813161 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:23.813158    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls podName:9f3f129d-c477-4771-a0cd-4c9e710bb832 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:39.813144709 +0000 UTC m=+67.029474571 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls") pod "dns-default-pvjjm" (UID: "9f3f129d-c477-4771-a0cd-4c9e710bb832") : secret "dns-default-metrics-tls" not found
Sep 09 05:50:25.123106 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:25.123065    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:25.123517 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:25.123203    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b784c5-0b83-493f-917e-559bc6a26b72-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:25.123517 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:25.123204    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:25.123517 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:50:25.123264    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls podName:23b784c5-0b83-493f-917e-559bc6a26b72 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:33.123250116 +0000 UTC m=+60.339579978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hc2v2" (UID: "23b784c5-0b83-493f-917e-559bc6a26b72") : secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:25.123753 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:25.123735    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b784c5-0b83-493f-917e-559bc6a26b72-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:28.118100 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:28.118067    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:28.118563 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:28.118128    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:28.118563 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:28.118526    2564 scope.go:117] "RemoveContainer" containerID="64b79b471744a4fd1c08e55701eaf818222cb57ce968076446adb4b7f2b9e640"
Sep 09 05:50:28.555936 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:28.555851    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 05:50:28.556077 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:28.555944    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-5mbcw" event={"ID":"207d756d-5ccd-456e-bd3f-3ede12b60f78","Type":"ContainerStarted","Data":"4d07ec9722894baff7fb1a862ee847feff5b642d644d681f1d4ea7b930c87496"}
Sep 09 05:50:28.556290 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:28.556258    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:28.574397 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:28.574342    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-5mbcw" podStartSLOduration=21.719305958 podStartE2EDuration="29.574328327s" totalImagesPullingTime="7.855022369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:28.573005939 +0000 UTC m=+55.789335823" watchObservedRunningTime="2026-09-09 05:50:28.574328327 +0000 UTC m=+55.790658274"
Sep 09 05:50:28.975483 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:28.975454    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-5mbcw"
Sep 09 05:50:33.197768 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:33.197739    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:33.199975 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:33.199955    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b784c5-0b83-493f-917e-559bc6a26b72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hc2v2\" (UID: \"23b784c5-0b83-493f-917e-559bc6a26b72\") " pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:33.357345 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:33.357316    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xqjvr\""
Sep 09 05:50:33.365183 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:33.365160    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hc2v2"
Sep 09 05:50:33.638156 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:33.638132    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hc2v2"]
Sep 09 05:50:33.640507 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:33.640474    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b784c5_0b83_493f_917e_559bc6a26b72.slice/crio-7c1a9250ac6c9222a4549eb84b5aa706add93cf5cba9745db876f9e104867e67 WatchSource:0}: Error finding container 7c1a9250ac6c9222a4549eb84b5aa706add93cf5cba9745db876f9e104867e67: Status 404 returned error can't find the container with id 7c1a9250ac6c9222a4549eb84b5aa706add93cf5cba9745db876f9e104867e67
Sep 09 05:50:34.575453 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:34.575419    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hc2v2" event={"ID":"23b784c5-0b83-493f-917e-559bc6a26b72","Type":"ContainerStarted","Data":"dd87b7418dc656cf3f236ea6410b5ca89aa3b1ea2be9ccb8a1b17728e18342c3"}
Sep 09 05:50:34.575815 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:34.575472    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hc2v2" event={"ID":"23b784c5-0b83-493f-917e-559bc6a26b72","Type":"ContainerStarted","Data":"31c2e14e277ca651f2797d2e3cce3dd7591b0127308f63c5e95cb4e7eaea8c19"}
Sep 09 05:50:34.575815 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:34.575509    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hc2v2" event={"ID":"23b784c5-0b83-493f-917e-559bc6a26b72","Type":"ContainerStarted","Data":"7c1a9250ac6c9222a4549eb84b5aa706add93cf5cba9745db876f9e104867e67"}
Sep 09 05:50:34.917816 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:34.917791    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm7sb"
Sep 09 05:50:36.584488 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:36.584453    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hc2v2" event={"ID":"23b784c5-0b83-493f-917e-559bc6a26b72","Type":"ContainerStarted","Data":"3a42938470ccd764e60215bd92cc0a3b3a5406d43b610200b2eb0e42a861f111"}
Sep 09 05:50:36.603917 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:36.603865    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hc2v2" podStartSLOduration=17.382124046 podStartE2EDuration="19.603850645s" totalImagesPullingTime="2.221726599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:50:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:36.602573271 +0000 UTC m=+63.818903158" watchObservedRunningTime="2026-09-09 05:50:36.603850645 +0000 UTC m=+63.820180529"
Sep 09 05:50:39.051223 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.051186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs\") pod \"network-metrics-daemon-qgb26\" (UID: \"910ab0f9-8007-4ec5-b5b9-bf0e32c85593\") " pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:50:39.051619 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.051282    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:50:39.054013 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.053995    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 05:50:39.054081 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.053995    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 05:50:39.064081 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.064054    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e1d1039-4eb0-4691-bd7f-21dfdd3870ca-original-pull-secret\") pod \"global-pull-secret-syncer-h8x9t\" (UID: \"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca\") " pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:50:39.064175 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.064105    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/910ab0f9-8007-4ec5-b5b9-bf0e32c85593-metrics-certs\") pod \"network-metrics-daemon-qgb26\" (UID: \"910ab0f9-8007-4ec5-b5b9-bf0e32c85593\") " pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:50:39.151834 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.151804    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrqj\" (UniqueName: \"kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj\") pod \"network-check-target-zkc2s\" (UID: \"4fb7e960-b0a8-4ba1-88b7-616e4910732a\") " pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:50:39.154147 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.154118    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkrqj\" (UniqueName: \"kubernetes.io/projected/4fb7e960-b0a8-4ba1-88b7-616e4910732a-kube-api-access-zkrqj\") pod \"network-check-target-zkc2s\" (UID: \"4fb7e960-b0a8-4ba1-88b7-616e4910732a\") " pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:50:39.209852 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.209827    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hh975\""
Sep 09 05:50:39.217765 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.217747    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:50:39.335398 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.335364    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8x9t"
Sep 09 05:50:39.338402 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.338384    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnkgt\""
Sep 09 05:50:39.346399 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.346382    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgb26"
Sep 09 05:50:39.491292 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.491265    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zkc2s"]
Sep 09 05:50:39.494398 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:39.494355    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb7e960_b0a8_4ba1_88b7_616e4910732a.slice/crio-c1bc3a1e89732801bf41d4f539e6695ad78830bcd3341473d2ec1b5b4c44adb7 WatchSource:0}: Error finding container c1bc3a1e89732801bf41d4f539e6695ad78830bcd3341473d2ec1b5b4c44adb7: Status 404 returned error can't find the container with id c1bc3a1e89732801bf41d4f539e6695ad78830bcd3341473d2ec1b5b4c44adb7
Sep 09 05:50:39.555209 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.555183    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:39.557392 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.557362    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls\") pod \"image-registry-64cdf68b7d-mdwmv\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") " pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:39.595457 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.595396    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zkc2s" event={"ID":"4fb7e960-b0a8-4ba1-88b7-616e4910732a","Type":"ContainerStarted","Data":"9713a3610684ac6446d2df2a67ea3f1b22dd9919f369f78434af4112dd8400d7"}
Sep 09 05:50:39.595457 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.595438    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zkc2s" event={"ID":"4fb7e960-b0a8-4ba1-88b7-616e4910732a","Type":"ContainerStarted","Data":"c1bc3a1e89732801bf41d4f539e6695ad78830bcd3341473d2ec1b5b4c44adb7"}
Sep 09 05:50:39.595629 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.595466    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:50:39.612618 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.612579    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zkc2s" podStartSLOduration=66.61256328 podStartE2EDuration="1m6.61256328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:39.611994149 +0000 UTC m=+66.828324034" watchObservedRunningTime="2026-09-09 05:50:39.61256328 +0000 UTC m=+66.828893163"
Sep 09 05:50:39.619646 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.619623    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h8x9t"]
Sep 09 05:50:39.622527 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:39.622504    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1d1039_4eb0_4691_bd7f_21dfdd3870ca.slice/crio-9fbac07050e6aeb03574a5fd26adeece3d2830d57f1967bec5e1e38111f904b2 WatchSource:0}: Error finding container 9fbac07050e6aeb03574a5fd26adeece3d2830d57f1967bec5e1e38111f904b2: Status 404 returned error can't find the container with id 9fbac07050e6aeb03574a5fd26adeece3d2830d57f1967bec5e1e38111f904b2
Sep 09 05:50:39.633211 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.633192    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qgb26"]
Sep 09 05:50:39.635951 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:39.635929    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910ab0f9_8007_4ec5_b5b9_bf0e32c85593.slice/crio-81839352061d698c8a9a17f532eb6e83038f27dafd99519fdd0f4e68c81214d7 WatchSource:0}: Error finding container 81839352061d698c8a9a17f532eb6e83038f27dafd99519fdd0f4e68c81214d7: Status 404 returned error can't find the container with id 81839352061d698c8a9a17f532eb6e83038f27dafd99519fdd0f4e68c81214d7
Sep 09 05:50:39.656601 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.656198    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jbrn2\" (UID: \"cf0f9476-232d-4dfb-8b60-175fba9847db\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"
Sep 09 05:50:39.656601 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.656316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:39.656601 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.656359    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:39.656601 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.656404    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7lfm4\" (UID: \"f163cf03-e312-4296-8fd6-cc95260b51b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4"
Sep 09 05:50:39.656601 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.656477    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-prgks\" (UID: \"f04421dd-ed42-46c3-997e-9bf3d4359d7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:39.659400 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.659367    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0f9476-232d-4dfb-8b60-175fba9847db-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jbrn2\" (UID: \"cf0f9476-232d-4dfb-8b60-175fba9847db\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"
Sep 09 05:50:39.660411 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.660012    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08804660-c36f-432a-80c1-6baba82c301f-service-ca-bundle\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:39.660411 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.660059    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f04421dd-ed42-46c3-997e-9bf3d4359d7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-prgks\" (UID: \"f04421dd-ed42-46c3-997e-9bf3d4359d7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:39.662136 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.662115    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08804660-c36f-432a-80c1-6baba82c301f-metrics-certs\") pod \"router-default-78695b457d-lfsxh\" (UID: \"08804660-c36f-432a-80c1-6baba82c301f\") " pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:39.662558 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.662541    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f163cf03-e312-4296-8fd6-cc95260b51b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7lfm4\" (UID: \"f163cf03-e312-4296-8fd6-cc95260b51b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4"
Sep 09 05:50:39.803014 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.802981    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gpg8k\""
Sep 09 05:50:39.810286 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.810261    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8zxvp\""
Sep 09 05:50:39.811307 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.811291    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:39.818286 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.818263    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"
Sep 09 05:50:39.828401 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.828384    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-j8bw9\""
Sep 09 05:50:39.836328 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.836297    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:39.838631 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.838613    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-96lk8\""
Sep 09 05:50:39.846414 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.846390    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4"
Sep 09 05:50:39.858185 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.858157    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:39.860806 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.860747    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f3f129d-c477-4771-a0cd-4c9e710bb832-metrics-tls\") pod \"dns-default-pvjjm\" (UID: \"9f3f129d-c477-4771-a0cd-4c9e710bb832\") " pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:39.871550 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.871305    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6rzjk\""
Sep 09 05:50:39.878959 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:39.878751    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"
Sep 09 05:50:40.010514 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.010466    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sfr25\""
Sep 09 05:50:40.018374 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.018351    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:40.093317 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.093214    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64cdf68b7d-mdwmv"]
Sep 09 05:50:40.094738 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.094708    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks"]
Sep 09 05:50:40.096409 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:40.096345    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cf9622_0adf_4129_955f_3e82431ab4ab.slice/crio-0f77db5c9a038493e49555bbf1d41adac7946c4cd21fd45d96fff11daf51be99 WatchSource:0}: Error finding container 0f77db5c9a038493e49555bbf1d41adac7946c4cd21fd45d96fff11daf51be99: Status 404 returned error can't find the container with id 0f77db5c9a038493e49555bbf1d41adac7946c4cd21fd45d96fff11daf51be99
Sep 09 05:50:40.097335 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:40.097317    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04421dd_ed42_46c3_997e_9bf3d4359d7a.slice/crio-09228dcae5a06c9d4530c6238f406b337a779801900d28b109db43bbe167adcf WatchSource:0}: Error finding container 09228dcae5a06c9d4530c6238f406b337a779801900d28b109db43bbe167adcf: Status 404 returned error can't find the container with id 09228dcae5a06c9d4530c6238f406b337a779801900d28b109db43bbe167adcf
Sep 09 05:50:40.136290 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.136262    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78695b457d-lfsxh"]
Sep 09 05:50:40.140219 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.140023    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-7lfm4"]
Sep 09 05:50:40.141281 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:40.141196    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08804660_c36f_432a_80c1_6baba82c301f.slice/crio-a257c3f7512d9ac0b069ecc3493f060e74897dfa6d6a16aef2b107a53380f3ab WatchSource:0}: Error finding container a257c3f7512d9ac0b069ecc3493f060e74897dfa6d6a16aef2b107a53380f3ab: Status 404 returned error can't find the container with id a257c3f7512d9ac0b069ecc3493f060e74897dfa6d6a16aef2b107a53380f3ab
Sep 09 05:50:40.143401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:40.143342    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf163cf03_e312_4296_8fd6_cc95260b51b3.slice/crio-369d5e5032e721dafc77f95a1942cb709db342e28f1025f57e9e34b8fe44fd7b WatchSource:0}: Error finding container 369d5e5032e721dafc77f95a1942cb709db342e28f1025f57e9e34b8fe44fd7b: Status 404 returned error can't find the container with id 369d5e5032e721dafc77f95a1942cb709db342e28f1025f57e9e34b8fe44fd7b
Sep 09 05:50:40.178159 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.178076    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2"]
Sep 09 05:50:40.288898 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.288808    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pvjjm"]
Sep 09 05:50:40.291836 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:40.291806    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3f129d_c477_4771_a0cd_4c9e710bb832.slice/crio-071b66bfc54c873d33b513dda7b4a106ccdd255e83ab6a4e5aeefcde67bfe2bb WatchSource:0}: Error finding container 071b66bfc54c873d33b513dda7b4a106ccdd255e83ab6a4e5aeefcde67bfe2bb: Status 404 returned error can't find the container with id 071b66bfc54c873d33b513dda7b4a106ccdd255e83ab6a4e5aeefcde67bfe2bb
Sep 09 05:50:40.601563 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.601521    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qgb26" event={"ID":"910ab0f9-8007-4ec5-b5b9-bf0e32c85593","Type":"ContainerStarted","Data":"81839352061d698c8a9a17f532eb6e83038f27dafd99519fdd0f4e68c81214d7"}
Sep 09 05:50:40.603508 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.603464    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h8x9t" event={"ID":"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca","Type":"ContainerStarted","Data":"9fbac07050e6aeb03574a5fd26adeece3d2830d57f1967bec5e1e38111f904b2"}
Sep 09 05:50:40.605675 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.605611    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvjjm" event={"ID":"9f3f129d-c477-4771-a0cd-4c9e710bb832","Type":"ContainerStarted","Data":"071b66bfc54c873d33b513dda7b4a106ccdd255e83ab6a4e5aeefcde67bfe2bb"}
Sep 09 05:50:40.607546 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.607515    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4" event={"ID":"f163cf03-e312-4296-8fd6-cc95260b51b3","Type":"ContainerStarted","Data":"369d5e5032e721dafc77f95a1942cb709db342e28f1025f57e9e34b8fe44fd7b"}
Sep 09 05:50:40.609452 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.609427    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78695b457d-lfsxh" event={"ID":"08804660-c36f-432a-80c1-6baba82c301f","Type":"ContainerStarted","Data":"dbe13aa4135d835418144bc663ac0134f20f9eac6056bb3bff99f89459ff27aa"}
Sep 09 05:50:40.609575 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.609470    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78695b457d-lfsxh" event={"ID":"08804660-c36f-432a-80c1-6baba82c301f","Type":"ContainerStarted","Data":"a257c3f7512d9ac0b069ecc3493f060e74897dfa6d6a16aef2b107a53380f3ab"}
Sep 09 05:50:40.611128 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.611101    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks" event={"ID":"f04421dd-ed42-46c3-997e-9bf3d4359d7a","Type":"ContainerStarted","Data":"09228dcae5a06c9d4530c6238f406b337a779801900d28b109db43bbe167adcf"}
Sep 09 05:50:40.612634 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.612605    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2" event={"ID":"cf0f9476-232d-4dfb-8b60-175fba9847db","Type":"ContainerStarted","Data":"a69f5d9356f15a2613bd7a6c2bd42b1850e7ee69e8eb9404aed014c689cd4eaa"}
Sep 09 05:50:40.615293 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.615266    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv" event={"ID":"48cf9622-0adf-4129-955f-3e82431ab4ab","Type":"ContainerStarted","Data":"87571929bfc050e373f6cc1a0cb96d715c1b4185e647406e2735f55d1f4fd9c3"}
Sep 09 05:50:40.615392 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.615318    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv" event={"ID":"48cf9622-0adf-4129-955f-3e82431ab4ab","Type":"ContainerStarted","Data":"0f77db5c9a038493e49555bbf1d41adac7946c4cd21fd45d96fff11daf51be99"}
Sep 09 05:50:40.634644 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.634555    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-78695b457d-lfsxh" podStartSLOduration=41.63453597 podStartE2EDuration="41.63453597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:40.632951556 +0000 UTC m=+67.849281440" watchObservedRunningTime="2026-09-09 05:50:40.63453597 +0000 UTC m=+67.850865850"
Sep 09 05:50:40.652991 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.652548    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv" podStartSLOduration=67.652527051 podStartE2EDuration="1m7.652527051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:40.651027174 +0000 UTC m=+67.867357060" watchObservedRunningTime="2026-09-09 05:50:40.652527051 +0000 UTC m=+67.868856936"
Sep 09 05:50:40.836712 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.836628    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:40.839840 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:40.839621    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:41.624699 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:41.624026    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qgb26" event={"ID":"910ab0f9-8007-4ec5-b5b9-bf0e32c85593","Type":"ContainerStarted","Data":"ead4866c5a5f8e15855346249fb0d00879b94e5b765f9ea5a4ce69dcd54919b0"}
Sep 09 05:50:41.624699 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:41.624090    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:41.624699 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:41.624661    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:50:41.626030 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:41.625826    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-78695b457d-lfsxh"
Sep 09 05:50:42.628935 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:42.628894    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qgb26" event={"ID":"910ab0f9-8007-4ec5-b5b9-bf0e32c85593","Type":"ContainerStarted","Data":"a8940f95d64353feaed32d5465a7b0bd63d8d7b35639d8ff00508e0410fc72b5"}
Sep 09 05:50:42.644635 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:42.644585    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qgb26" podStartSLOduration=67.98009336 podStartE2EDuration="1m9.644568862s" totalImagesPullingTime="1.664475502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:42.642573636 +0000 UTC m=+69.858903521" watchObservedRunningTime="2026-09-09 05:50:42.644568862 +0000 UTC m=+69.860898746"
Sep 09 05:50:45.644350 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:45.644268    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks" event={"ID":"f04421dd-ed42-46c3-997e-9bf3d4359d7a","Type":"ContainerStarted","Data":"53876c0d1190ff20ead99aafbad816b44871bd6fe291adff0aeb46de13e4b287"}
Sep 09 05:50:45.646779 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:45.646591    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2" event={"ID":"cf0f9476-232d-4dfb-8b60-175fba9847db","Type":"ContainerStarted","Data":"f9fe5340cc126048b096e8f4223ed601e2474583d2cdcfe790c921ee94971bd8"}
Sep 09 05:50:45.650866 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:45.650841    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvjjm" event={"ID":"9f3f129d-c477-4771-a0cd-4c9e710bb832","Type":"ContainerStarted","Data":"2ca64ea086776241e71b5bf758117a94846ae7198fcdddaa90d3adfef82f028b"}
Sep 09 05:50:45.652868 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:45.652839    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4" event={"ID":"f163cf03-e312-4296-8fd6-cc95260b51b3","Type":"ContainerStarted","Data":"5d11fdf42a02f53d4d5a3bf871e92528df27cb908e0824603422dd2f95ef33ad"}
Sep 09 05:50:45.690738 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:45.690679    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-prgks" podStartSLOduration=41.367386653 podStartE2EDuration="46.690658326s" totalImagesPullingTime="5.323271673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:45.689582705 +0000 UTC m=+72.905912590" watchObservedRunningTime="2026-09-09 05:50:45.690658326 +0000 UTC m=+72.906988213"
Sep 09 05:50:46.534535 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.534453    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-7lfm4" podStartSLOduration=42.303668628 podStartE2EDuration="47.534430595s" totalImagesPullingTime="5.230761967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:45.707836211 +0000 UTC m=+72.924166096" watchObservedRunningTime="2026-09-09 05:50:46.534430595 +0000 UTC m=+73.750760479"
Sep 09 05:50:46.535841 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.535815    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5979cf6c99-xlfkv"]
Sep 09 05:50:46.552822 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.552791    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5979cf6c99-xlfkv"]
Sep 09 05:50:46.552938 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.552902    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.555526 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.555503    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-przmp\""
Sep 09 05:50:46.555668 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.555503    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 05:50:46.555668 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.555558    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 05:50:46.555668 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.555564    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 05:50:46.556391 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.556366    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 05:50:46.556391 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.556381    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 05:50:46.556547 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.556411    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 05:50:46.556547 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.556413    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 05:50:46.657787 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.657753    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvjjm" event={"ID":"9f3f129d-c477-4771-a0cd-4c9e710bb832","Type":"ContainerStarted","Data":"0718adeae5b548fd8f9079c5e3a288311de8e0c732cc4ce39c1dc39626c0c245"}
Sep 09 05:50:46.658192 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.657853    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:46.659563 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.659528    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2" event={"ID":"cf0f9476-232d-4dfb-8b60-175fba9847db","Type":"ContainerStarted","Data":"987cad768b0aec3d81f8ab254c84090c721018e85df450ede76aff5d4c626b71"}
Sep 09 05:50:46.661078 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.661053    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h8x9t" event={"ID":"9e1d1039-4eb0-4691-bd7f-21dfdd3870ca","Type":"ContainerStarted","Data":"f994679d63a52eb3c6c1723b9a7236ec647485a50c251eeb8e5a64c078daf3d9"}
Sep 09 05:50:46.674647 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.674608    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pvjjm" podStartSLOduration=34.546003902 podStartE2EDuration="39.674595301s" totalImagesPullingTime="5.128591399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:46.673594491 +0000 UTC m=+73.889924390" watchObservedRunningTime="2026-09-09 05:50:46.674595301 +0000 UTC m=+73.890925206"
Sep 09 05:50:46.687720 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.687682    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h8x9t" podStartSLOduration=67.876493913 podStartE2EDuration="1m13.687672305s" totalImagesPullingTime="5.811178392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:46.687257731 +0000 UTC m=+73.903587615" watchObservedRunningTime="2026-09-09 05:50:46.687672305 +0000 UTC m=+73.904002189"
Sep 09 05:50:46.705379 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.705326    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jbrn2" podStartSLOduration=42.506893658 podStartE2EDuration="47.70531269s" totalImagesPullingTime="5.198419032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:46.704896497 +0000 UTC m=+73.921226380" watchObservedRunningTime="2026-09-09 05:50:46.70531269 +0000 UTC m=+73.921642574"
Sep 09 05:50:46.713419 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.713396    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-service-ca\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.713542 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.713445    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-serving-cert\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.713542 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.713461    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdspw\" (UniqueName: \"kubernetes.io/projected/72b883f9-df2b-4c15-ab55-79a6b927cd97-kube-api-access-cdspw\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.713630 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.713557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-config\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.713630 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.713607    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-oauth-config\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.713713 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.713630    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-oauth-serving-cert\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.814812 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.814740    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-serving-cert\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.814812 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.814769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdspw\" (UniqueName: \"kubernetes.io/projected/72b883f9-df2b-4c15-ab55-79a6b927cd97-kube-api-access-cdspw\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.814812 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.814802    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-config\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.815051 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.814894    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-oauth-config\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.815051 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.814920    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-oauth-serving-cert\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.815051 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.814965    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-service-ca\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.816313 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.816289    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-service-ca\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.816391 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.816321    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-oauth-serving-cert\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.816391 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.816340    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-config\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.817447 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.817426    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-oauth-config\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.817685 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.817666    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-serving-cert\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.822779 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.822756    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdspw\" (UniqueName: \"kubernetes.io/projected/72b883f9-df2b-4c15-ab55-79a6b927cd97-kube-api-access-cdspw\") pod \"console-5979cf6c99-xlfkv\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") " pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:46.862787 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:46.862747    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:47.159183 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:47.159149    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5979cf6c99-xlfkv"]
Sep 09 05:50:47.162867 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:47.162843    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b883f9_df2b_4c15_ab55_79a6b927cd97.slice/crio-5ef73391f3ab84d76a1cf425f2a607f3f868a82270f9190e9cd5b8777f3dd693 WatchSource:0}: Error finding container 5ef73391f3ab84d76a1cf425f2a607f3f868a82270f9190e9cd5b8777f3dd693: Status 404 returned error can't find the container with id 5ef73391f3ab84d76a1cf425f2a607f3f868a82270f9190e9cd5b8777f3dd693
Sep 09 05:50:47.666153 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:47.666113    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5979cf6c99-xlfkv" event={"ID":"72b883f9-df2b-4c15-ab55-79a6b927cd97","Type":"ContainerStarted","Data":"5ef73391f3ab84d76a1cf425f2a607f3f868a82270f9190e9cd5b8777f3dd693"}
Sep 09 05:50:50.680994 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:50.680914    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5979cf6c99-xlfkv" event={"ID":"72b883f9-df2b-4c15-ab55-79a6b927cd97","Type":"ContainerStarted","Data":"a4f1c58d5425a9294f1d465a1142173ea04d9dd81015dd45cc1900f93308ff1f"}
Sep 09 05:50:50.697855 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:50.697808    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5979cf6c99-xlfkv" podStartSLOduration=1.4671100830000001 podStartE2EDuration="4.697793887s" totalImagesPullingTime="3.230683804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:50:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:50.696736269 +0000 UTC m=+77.913066156" watchObservedRunningTime="2026-09-09 05:50:50.697793887 +0000 UTC m=+77.914123770"
Sep 09 05:50:53.050161 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.050128    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-559c47d9b5-rrcbc"]
Sep 09 05:50:53.053141 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.053122    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.060950 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.060925    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 05:50:53.063527 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.063508    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-559c47d9b5-rrcbc"]
Sep 09 05:50:53.167917 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.167886    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-trusted-ca-bundle\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.168059 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.167927    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-serving-cert\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.168059 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.167964    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-oauth-serving-cert\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.168059 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.167984    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-config\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.168059 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.168018    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-service-ca\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.168059 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.168059    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwjht\" (UniqueName: \"kubernetes.io/projected/845be81c-3808-4bd3-90bb-fc9b3a4a198c-kube-api-access-xwjht\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.168277 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.168143    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-oauth-config\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.269124 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.269081    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-oauth-config\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.269314 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.269143    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-trusted-ca-bundle\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.269314 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.269174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-serving-cert\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.269314 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.269223    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-oauth-serving-cert\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.269314 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.269251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-config\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.269314 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.269273    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-service-ca\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.269314 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.269297    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjht\" (UniqueName: \"kubernetes.io/projected/845be81c-3808-4bd3-90bb-fc9b3a4a198c-kube-api-access-xwjht\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.270088 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.270053    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-config\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.270221 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.270111    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-service-ca\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.270221 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.270135    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-oauth-serving-cert\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.270338 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.270255    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-trusted-ca-bundle\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.271677 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.271656    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-oauth-config\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.271834 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.271817    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-serving-cert\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.277615 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.277594    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjht\" (UniqueName: \"kubernetes.io/projected/845be81c-3808-4bd3-90bb-fc9b3a4a198c-kube-api-access-xwjht\") pod \"console-559c47d9b5-rrcbc\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") " pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.362579 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.362548    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:50:53.633093 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.633069    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-559c47d9b5-rrcbc"]
Sep 09 05:50:53.635471 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:50:53.635441    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845be81c_3808_4bd3_90bb_fc9b3a4a198c.slice/crio-549cac575b8fba1748bec4362fa3bfbfec843dff73b81d0e4b3d4b826a328980 WatchSource:0}: Error finding container 549cac575b8fba1748bec4362fa3bfbfec843dff73b81d0e4b3d4b826a328980: Status 404 returned error can't find the container with id 549cac575b8fba1748bec4362fa3bfbfec843dff73b81d0e4b3d4b826a328980
Sep 09 05:50:53.693836 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:53.693808    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559c47d9b5-rrcbc" event={"ID":"845be81c-3808-4bd3-90bb-fc9b3a4a198c","Type":"ContainerStarted","Data":"549cac575b8fba1748bec4362fa3bfbfec843dff73b81d0e4b3d4b826a328980"}
Sep 09 05:50:54.699080 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:54.699049    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559c47d9b5-rrcbc" event={"ID":"845be81c-3808-4bd3-90bb-fc9b3a4a198c","Type":"ContainerStarted","Data":"75dbb3f49cd0654305f5467f1b60fbc1300a127ae5edec3ad0a6bbac69cfeb7a"}
Sep 09 05:50:54.717141 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:54.717094    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-559c47d9b5-rrcbc" podStartSLOduration=1.717078975 podStartE2EDuration="1.717078975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:54.716033709 +0000 UTC m=+81.932363593" watchObservedRunningTime="2026-09-09 05:50:54.717078975 +0000 UTC m=+81.933408859"
Sep 09 05:50:56.668995 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:56.668881    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pvjjm"
Sep 09 05:50:56.863124 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:56.863090    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:56.863299 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:56.863278    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:56.867838 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:56.867816    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:50:57.715849 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:50:57.715820    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:51:01.803558 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:01.803526    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64cdf68b7d-mdwmv"]
Sep 09 05:51:01.807836 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:01.807813    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:51:03.362661 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:03.362622    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:51:03.363134 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:03.362685    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:51:03.367616 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:03.367582    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:51:03.738283 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:03.738207    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:51:03.782192 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:03.782159    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5979cf6c99-xlfkv"]
Sep 09 05:51:04.516327 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.516290    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"]
Sep 09 05:51:04.520818 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.520796    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"
Sep 09 05:51:04.523178 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.523153    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qrqv8\""
Sep 09 05:51:04.523445 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.523430    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 05:51:04.524092 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.524074    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 05:51:04.524092 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.524082    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 05:51:04.548550 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.548521    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"]
Sep 09 05:51:04.578165 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.578137    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8kmtm"]
Sep 09 05:51:04.581326 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.581311    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.585040 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.585021    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 05:51:04.585921 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.585907    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 05:51:04.586354 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.586337    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8mwq7\""
Sep 09 05:51:04.586412 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.586379    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 05:51:04.660196 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.660163    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlnw2\" (UniqueName: \"kubernetes.io/projected/1b589342-531d-4661-8840-f9a74a79ae59-kube-api-access-hlnw2\") pod \"openshift-state-metrics-657fd6c54-hmsjp\" (UID: \"1b589342-531d-4661-8840-f9a74a79ae59\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"
Sep 09 05:51:04.660344 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.660202    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b589342-531d-4661-8840-f9a74a79ae59-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-hmsjp\" (UID: \"1b589342-531d-4661-8840-f9a74a79ae59\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"
Sep 09 05:51:04.660344 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.660220    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b589342-531d-4661-8840-f9a74a79ae59-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-hmsjp\" (UID: \"1b589342-531d-4661-8840-f9a74a79ae59\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"
Sep 09 05:51:04.660344 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.660242    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b589342-531d-4661-8840-f9a74a79ae59-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-hmsjp\" (UID: \"1b589342-531d-4661-8840-f9a74a79ae59\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"
Sep 09 05:51:04.760961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.760924    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b589342-531d-4661-8840-f9a74a79ae59-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-hmsjp\" (UID: \"1b589342-531d-4661-8840-f9a74a79ae59\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"
Sep 09 05:51:04.760961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.760964    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-tls\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.761148 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.760983    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ltjn\" (UniqueName: \"kubernetes.io/projected/351a845d-670b-465d-a3ce-2ac23704a55d-kube-api-access-7ltjn\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.761148 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.761008    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.761148 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.761099    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.761148 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.761137    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/351a845d-670b-465d-a3ce-2ac23704a55d-metrics-client-ca\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.761273 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.761190    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/351a845d-670b-465d-a3ce-2ac23704a55d-root\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.761273 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.761214    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-textfile\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.761273 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.761248    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-wtmp\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.761364 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.761302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlnw2\" (UniqueName: \"kubernetes.io/projected/1b589342-531d-4661-8840-f9a74a79ae59-kube-api-access-hlnw2\") pod \"openshift-state-metrics-657fd6c54-hmsjp\" (UID: \"1b589342-531d-4661-8840-f9a74a79ae59\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"
Sep 09 05:51:04.761364 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.761336    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/351a845d-670b-465d-a3ce-2ac23704a55d-sys\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.761538 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.761470    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b589342-531d-4661-8840-f9a74a79ae59-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-hmsjp\" (UID: \"1b589342-531d-4661-8840-f9a74a79ae59\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"
Sep 09 05:51:04.761538 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.761532    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b589342-531d-4661-8840-f9a74a79ae59-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-hmsjp\" (UID: \"1b589342-531d-4661-8840-f9a74a79ae59\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"
Sep 09 05:51:04.761653 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:51:04.761569    2564 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 09 05:51:04.761653 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:51:04.761621    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b589342-531d-4661-8840-f9a74a79ae59-openshift-state-metrics-tls podName:1b589342-531d-4661-8840-f9a74a79ae59 nodeName:}" failed. No retries permitted until 2026-09-09 05:51:05.261604984 +0000 UTC m=+92.477934846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1b589342-531d-4661-8840-f9a74a79ae59-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-hmsjp" (UID: "1b589342-531d-4661-8840-f9a74a79ae59") : secret "openshift-state-metrics-tls" not found
Sep 09 05:51:04.762065 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.762046    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b589342-531d-4661-8840-f9a74a79ae59-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-hmsjp\" (UID: \"1b589342-531d-4661-8840-f9a74a79ae59\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"
Sep 09 05:51:04.763375 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.763358    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b589342-531d-4661-8840-f9a74a79ae59-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-hmsjp\" (UID: \"1b589342-531d-4661-8840-f9a74a79ae59\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"
Sep 09 05:51:04.773747 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.773693    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlnw2\" (UniqueName: \"kubernetes.io/projected/1b589342-531d-4661-8840-f9a74a79ae59-kube-api-access-hlnw2\") pod \"openshift-state-metrics-657fd6c54-hmsjp\" (UID: \"1b589342-531d-4661-8840-f9a74a79ae59\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"
Sep 09 05:51:04.862536 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.862505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-wtmp\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.862676 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.862542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/351a845d-670b-465d-a3ce-2ac23704a55d-sys\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.862676 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.862603    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-tls\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.862676 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.862622    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ltjn\" (UniqueName: \"kubernetes.io/projected/351a845d-670b-465d-a3ce-2ac23704a55d-kube-api-access-7ltjn\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.862676 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.862638    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.862871 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.862675    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-wtmp\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.862871 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.862705    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.862871 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:51:04.862767    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 05:51:04.862871 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.862825    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/351a845d-670b-465d-a3ce-2ac23704a55d-sys\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.862871 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.862769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/351a845d-670b-465d-a3ce-2ac23704a55d-metrics-client-ca\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.862871 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:51:04.862831    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-tls podName:351a845d-670b-465d-a3ce-2ac23704a55d nodeName:}" failed. No retries permitted until 2026-09-09 05:51:05.362810923 +0000 UTC m=+92.579140801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-tls") pod "node-exporter-8kmtm" (UID: "351a845d-670b-465d-a3ce-2ac23704a55d") : secret "node-exporter-tls" not found
Sep 09 05:51:04.863156 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.863097    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/351a845d-670b-465d-a3ce-2ac23704a55d-root\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.863156 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.863129    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-textfile\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.863250 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.863162    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/351a845d-670b-465d-a3ce-2ac23704a55d-root\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.863476 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.863452    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/351a845d-670b-465d-a3ce-2ac23704a55d-metrics-client-ca\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.863558 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.863456    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.863558 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.863504    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-textfile\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.865452 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.865427    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:04.872902 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:04.872884    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ltjn\" (UniqueName: \"kubernetes.io/projected/351a845d-670b-465d-a3ce-2ac23704a55d-kube-api-access-7ltjn\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:05.266818 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.266783    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b589342-531d-4661-8840-f9a74a79ae59-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-hmsjp\" (UID: \"1b589342-531d-4661-8840-f9a74a79ae59\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"
Sep 09 05:51:05.269158 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.269132    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b589342-531d-4661-8840-f9a74a79ae59-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-hmsjp\" (UID: \"1b589342-531d-4661-8840-f9a74a79ae59\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"
Sep 09 05:51:05.368082 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.368049    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-tls\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:05.371405 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.371361    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/351a845d-670b-465d-a3ce-2ac23704a55d-node-exporter-tls\") pod \"node-exporter-8kmtm\" (UID: \"351a845d-670b-465d-a3ce-2ac23704a55d\") " pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:05.430956 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.430924    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"
Sep 09 05:51:05.490627 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.490596    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8kmtm"
Sep 09 05:51:05.504983 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:51:05.504958    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351a845d_670b_465d_a3ce_2ac23704a55d.slice/crio-6b208cf608b3bb94abf2fa4d81069df58669d5e6a52b8a153235876b8a441e03 WatchSource:0}: Error finding container 6b208cf608b3bb94abf2fa4d81069df58669d5e6a52b8a153235876b8a441e03: Status 404 returned error can't find the container with id 6b208cf608b3bb94abf2fa4d81069df58669d5e6a52b8a153235876b8a441e03
Sep 09 05:51:05.526879 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.526816    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 05:51:05.530711 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.530694    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.533380 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.533306    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 05:51:05.533380 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.533346    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 05:51:05.533380 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.533364    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 05:51:05.533618 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.533400    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cftx8\""
Sep 09 05:51:05.533618 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.533367    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 05:51:05.533618 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.533347    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 05:51:05.533814 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.533695    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 05:51:05.533814 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.533733    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 05:51:05.533814 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.533780    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 05:51:05.533814 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.533791    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 05:51:05.544536 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.544514    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 05:51:05.671766 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.671727    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.671932 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.671777    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.671932 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.671854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5d98f6c8-6c49-44df-8610-2583c98be150-config-out\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.671932 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.671901    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d98f6c8-6c49-44df-8610-2583c98be150-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.672068 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.671928    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnbxx\" (UniqueName: \"kubernetes.io/projected/5d98f6c8-6c49-44df-8610-2583c98be150-kube-api-access-fnbxx\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.672068 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.671968    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-config-volume\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.672068 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.671991    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5d98f6c8-6c49-44df-8610-2583c98be150-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.672068 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.672015    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.672068 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.672057    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.672234 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.672088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-web-config\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.672234 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.672116    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5d98f6c8-6c49-44df-8610-2583c98be150-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.672234 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.672185    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d98f6c8-6c49-44df-8610-2583c98be150-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.672234 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.672215    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.701873 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.701852    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp"]
Sep 09 05:51:05.704091 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:51:05.704061    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b589342_531d_4661_8840_f9a74a79ae59.slice/crio-6d3bf09c891c9e0e1331e10ea50cee42e8971d85523b46bb2274c1a83a583a25 WatchSource:0}: Error finding container 6d3bf09c891c9e0e1331e10ea50cee42e8971d85523b46bb2274c1a83a583a25: Status 404 returned error can't find the container with id 6d3bf09c891c9e0e1331e10ea50cee42e8971d85523b46bb2274c1a83a583a25
Sep 09 05:51:05.744213 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.744185    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp" event={"ID":"1b589342-531d-4661-8840-f9a74a79ae59","Type":"ContainerStarted","Data":"6d3bf09c891c9e0e1331e10ea50cee42e8971d85523b46bb2274c1a83a583a25"}
Sep 09 05:51:05.745515 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.745470    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kmtm" event={"ID":"351a845d-670b-465d-a3ce-2ac23704a55d","Type":"ContainerStarted","Data":"6b208cf608b3bb94abf2fa4d81069df58669d5e6a52b8a153235876b8a441e03"}
Sep 09 05:51:05.773513 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.773468    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnbxx\" (UniqueName: \"kubernetes.io/projected/5d98f6c8-6c49-44df-8610-2583c98be150-kube-api-access-fnbxx\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.773631 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.773542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-config-volume\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.773631 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.773566    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5d98f6c8-6c49-44df-8610-2583c98be150-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.773631 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.773591    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.773781 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.773634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.773781 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.773669    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-web-config\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.773781 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.773696    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5d98f6c8-6c49-44df-8610-2583c98be150-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.773781 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.773731    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d98f6c8-6c49-44df-8610-2583c98be150-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.773781 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.773762    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.774019 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.773806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.774019 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.773839    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.774019 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.773877    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5d98f6c8-6c49-44df-8610-2583c98be150-config-out\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.774019 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.773904    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d98f6c8-6c49-44df-8610-2583c98be150-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.774451 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.774421    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5d98f6c8-6c49-44df-8610-2583c98be150-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.774674 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.774652    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d98f6c8-6c49-44df-8610-2583c98be150-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.776251 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.776222    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d98f6c8-6c49-44df-8610-2583c98be150-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.778430 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.778337    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.779196 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.779104    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-config-volume\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.779196 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.779137    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.779340 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.779275    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5d98f6c8-6c49-44df-8610-2583c98be150-config-out\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.779975 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.779932    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.780449 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.780355    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5d98f6c8-6c49-44df-8610-2583c98be150-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.780543 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.780476    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.780847 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.780826    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.781248 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.781227    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-web-config\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.783069 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.783052    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnbxx\" (UniqueName: \"kubernetes.io/projected/5d98f6c8-6c49-44df-8610-2583c98be150-kube-api-access-fnbxx\") pod \"alertmanager-main-0\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:05.840317 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:05.840294    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:51:06.122727 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:06.122696    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 05:51:06.125112 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:51:06.125080    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d98f6c8_6c49_44df_8610_2583c98be150.slice/crio-ed5a5cf04fcc4785208885d6d923377cb606e3bf0727ee4e7039718b8daf1b1f WatchSource:0}: Error finding container ed5a5cf04fcc4785208885d6d923377cb606e3bf0727ee4e7039718b8daf1b1f: Status 404 returned error can't find the container with id ed5a5cf04fcc4785208885d6d923377cb606e3bf0727ee4e7039718b8daf1b1f
Sep 09 05:51:06.752459 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:06.752352    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5d98f6c8-6c49-44df-8610-2583c98be150","Type":"ContainerStarted","Data":"ed5a5cf04fcc4785208885d6d923377cb606e3bf0727ee4e7039718b8daf1b1f"}
Sep 09 05:51:06.755570 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:06.755449    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp" event={"ID":"1b589342-531d-4661-8840-f9a74a79ae59","Type":"ContainerStarted","Data":"3630ad65ccbcdc9b9d5fdc2a2bc169863a912c89e36c4d86318e61a814d55b51"}
Sep 09 05:51:06.755570 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:06.755522    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp" event={"ID":"1b589342-531d-4661-8840-f9a74a79ae59","Type":"ContainerStarted","Data":"b17eaca977416e782694164db7ef31d521d5120905e7d44d07f032a5e681078c"}
Sep 09 05:51:07.761711 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:07.761668    2564 generic.go:356] "Generic (PLEG): container finished" podID="351a845d-670b-465d-a3ce-2ac23704a55d" containerID="a0f7370631db5472433e7821cbf5bce006a5b6cb8a68ff37b1fbff53fd5332cf" exitCode=0
Sep 09 05:51:07.762153 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:07.761747    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kmtm" event={"ID":"351a845d-670b-465d-a3ce-2ac23704a55d","Type":"ContainerDied","Data":"a0f7370631db5472433e7821cbf5bce006a5b6cb8a68ff37b1fbff53fd5332cf"}
Sep 09 05:51:07.763380 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:07.763358    2564 generic.go:356] "Generic (PLEG): container finished" podID="5d98f6c8-6c49-44df-8610-2583c98be150" containerID="9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454" exitCode=0
Sep 09 05:51:07.763487 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:07.763393    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5d98f6c8-6c49-44df-8610-2583c98be150","Type":"ContainerDied","Data":"9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454"}
Sep 09 05:51:07.765953 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:07.765920    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp" event={"ID":"1b589342-531d-4661-8840-f9a74a79ae59","Type":"ContainerStarted","Data":"a7700c552cd56cc95771a0ab16b02199c7f85ec7e90d8572c44e4255bad71b31"}
Sep 09 05:51:07.833353 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:07.833055    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hmsjp" podStartSLOduration=2.75895984 podStartE2EDuration="3.833036323s" totalImagesPullingTime="1.074076483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:51:07.832670507 +0000 UTC m=+95.049000392" watchObservedRunningTime="2026-09-09 05:51:07.833036323 +0000 UTC m=+95.049366208"
Sep 09 05:51:08.772855 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:08.772803    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kmtm" event={"ID":"351a845d-670b-465d-a3ce-2ac23704a55d","Type":"ContainerStarted","Data":"cc258efe52d64978e93521fdf136a55fee0f2b112471b890a415b17846d3e417"}
Sep 09 05:51:08.773304 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:08.772873    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kmtm" event={"ID":"351a845d-670b-465d-a3ce-2ac23704a55d","Type":"ContainerStarted","Data":"16a51147af3698a63a2f5c54b21e663498fd24acaa5f93fd53331366e227d8ea"}
Sep 09 05:51:08.794616 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:08.794559    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8kmtm" podStartSLOduration=3.3622053960000002 podStartE2EDuration="4.79453909s" totalImagesPullingTime="1.360555825s" totalInitContainerRuntime="71.777869ms" isStatefulPod=true podCreationTimestamp="2026-09-09 05:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:51:08.792741713 +0000 UTC m=+96.009071619" watchObservedRunningTime="2026-09-09 05:51:08.79453909 +0000 UTC m=+96.010868977"
Sep 09 05:51:09.241896 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.241863    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dc9546fb7-s84mv"]
Sep 09 05:51:09.244985 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.244955    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.258121 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.258094    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dc9546fb7-s84mv"]
Sep 09 05:51:09.308590 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.308560    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-serving-cert\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.308709 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.308607    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-oauth-serving-cert\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.308709 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.308644    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-trusted-ca-bundle\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.308709 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.308676    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm74k\" (UniqueName: \"kubernetes.io/projected/d1ff1aaa-b886-434e-8c4a-06f73f033576-kube-api-access-mm74k\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.308709 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.308701    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-config\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.308895 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.308761    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-oauth-config\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.308895 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.308840    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-service-ca\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.409633 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.409609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-service-ca\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.409768 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.409663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-serving-cert\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.409768 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.409684    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-oauth-serving-cert\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.409768 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.409706    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-trusted-ca-bundle\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.409768 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.409724    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm74k\" (UniqueName: \"kubernetes.io/projected/d1ff1aaa-b886-434e-8c4a-06f73f033576-kube-api-access-mm74k\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.409768 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.409741    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-config\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.409768 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.409783    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-oauth-config\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.410381 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.410354    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-service-ca\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.410468 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.410424    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-oauth-serving-cert\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.410739 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.410713    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-trusted-ca-bundle\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.410896 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.410871    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-config\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.412168 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.412145    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-serving-cert\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.412874 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.412853    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-oauth-config\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.418053 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.418030    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm74k\" (UniqueName: \"kubernetes.io/projected/d1ff1aaa-b886-434e-8c4a-06f73f033576-kube-api-access-mm74k\") pod \"console-5dc9546fb7-s84mv\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") " pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.555981 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.555953    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:09.782351 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.782266    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5d98f6c8-6c49-44df-8610-2583c98be150","Type":"ContainerStarted","Data":"4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90"}
Sep 09 05:51:09.782716 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.782376    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5d98f6c8-6c49-44df-8610-2583c98be150","Type":"ContainerStarted","Data":"f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d"}
Sep 09 05:51:09.782716 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.782422    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5d98f6c8-6c49-44df-8610-2583c98be150","Type":"ContainerStarted","Data":"f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e"}
Sep 09 05:51:09.782716 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.782460    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5d98f6c8-6c49-44df-8610-2583c98be150","Type":"ContainerStarted","Data":"0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2"}
Sep 09 05:51:09.782716 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.782488    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5d98f6c8-6c49-44df-8610-2583c98be150","Type":"ContainerStarted","Data":"ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0"}
Sep 09 05:51:09.837970 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:09.837939    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dc9546fb7-s84mv"]
Sep 09 05:51:09.841030 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:51:09.841007    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ff1aaa_b886_434e_8c4a_06f73f033576.slice/crio-c73cf8b49127ae3a016fb10564d469bcfed2202725dffaf2893fd8b35c5201da WatchSource:0}: Error finding container c73cf8b49127ae3a016fb10564d469bcfed2202725dffaf2893fd8b35c5201da: Status 404 returned error can't find the container with id c73cf8b49127ae3a016fb10564d469bcfed2202725dffaf2893fd8b35c5201da
Sep 09 05:51:10.618636 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:10.618599    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zkc2s"
Sep 09 05:51:10.789912 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:10.789878    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5d98f6c8-6c49-44df-8610-2583c98be150","Type":"ContainerStarted","Data":"a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb"}
Sep 09 05:51:10.791526 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:10.791476    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dc9546fb7-s84mv" event={"ID":"d1ff1aaa-b886-434e-8c4a-06f73f033576","Type":"ContainerStarted","Data":"e7175a0f68898300af7402bf35e154944a84be742e3c7456201f013eabb26b23"}
Sep 09 05:51:10.791626 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:10.791544    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dc9546fb7-s84mv" event={"ID":"d1ff1aaa-b886-434e-8c4a-06f73f033576","Type":"ContainerStarted","Data":"c73cf8b49127ae3a016fb10564d469bcfed2202725dffaf2893fd8b35c5201da"}
Sep 09 05:51:10.820717 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:10.820668    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.026496925 podStartE2EDuration="5.820648076s" totalImagesPullingTime="3.775170376s" totalInitContainerRuntime="18.980775ms" isStatefulPod=false podCreationTimestamp="2026-09-09 05:51:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:51:10.818617801 +0000 UTC m=+98.034947685" watchObservedRunningTime="2026-09-09 05:51:10.820648076 +0000 UTC m=+98.036977963"
Sep 09 05:51:10.839986 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:10.839939    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5dc9546fb7-s84mv" podStartSLOduration=1.839924528 podStartE2EDuration="1.839924528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:51:10.83885809 +0000 UTC m=+98.055187975" watchObservedRunningTime="2026-09-09 05:51:10.839924528 +0000 UTC m=+98.056254411"
Sep 09 05:51:13.678245 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.678209    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dc9546fb7-s84mv"]
Sep 09 05:51:13.716025 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.715992    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cd8996fdc-chl5s"]
Sep 09 05:51:13.722753 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.722726    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.727935 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.727908    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd8996fdc-chl5s"]
Sep 09 05:51:13.749099 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.749071    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33277503-645e-402a-9629-c689c9a2df1f-console-serving-cert\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.749099 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.749098    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-oauth-serving-cert\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.749258 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.749120    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-trusted-ca-bundle\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.749258 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.749146    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg86p\" (UniqueName: \"kubernetes.io/projected/33277503-645e-402a-9629-c689c9a2df1f-kube-api-access-fg86p\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.749258 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.749171    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-console-config\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.749258 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.749222    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-service-ca\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.749374 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.749264    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33277503-645e-402a-9629-c689c9a2df1f-console-oauth-config\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.849941 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.849907    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33277503-645e-402a-9629-c689c9a2df1f-console-oauth-config\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.850081 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.849966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33277503-645e-402a-9629-c689c9a2df1f-console-serving-cert\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.850081 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.849985    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-oauth-serving-cert\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.850081 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.850008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-trusted-ca-bundle\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.850081 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.850033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg86p\" (UniqueName: \"kubernetes.io/projected/33277503-645e-402a-9629-c689c9a2df1f-kube-api-access-fg86p\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.850308 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.850192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-console-config\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.850308 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.850246    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-service-ca\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.850869 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.850845    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-oauth-serving-cert\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.851024 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.851000    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-console-config\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.851069 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.851015    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-service-ca\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.851069 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.851031    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-trusted-ca-bundle\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.852397 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.852365    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33277503-645e-402a-9629-c689c9a2df1f-console-oauth-config\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.852536 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.852514    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33277503-645e-402a-9629-c689c9a2df1f-console-serving-cert\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:13.857134 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:13.857114    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg86p\" (UniqueName: \"kubernetes.io/projected/33277503-645e-402a-9629-c689c9a2df1f-kube-api-access-fg86p\") pod \"console-5cd8996fdc-chl5s\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") " pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:14.032571 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:14.032469    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:14.309533 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:14.309424    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd8996fdc-chl5s"]
Sep 09 05:51:14.312224 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:51:14.312199    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33277503_645e_402a_9629_c689c9a2df1f.slice/crio-f2f544c65fd5ee9bb8d3fffce25ea1ee89f0d831c263bd81c72d397d8cae2b1d WatchSource:0}: Error finding container f2f544c65fd5ee9bb8d3fffce25ea1ee89f0d831c263bd81c72d397d8cae2b1d: Status 404 returned error can't find the container with id f2f544c65fd5ee9bb8d3fffce25ea1ee89f0d831c263bd81c72d397d8cae2b1d
Sep 09 05:51:14.809929 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:14.809897    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd8996fdc-chl5s" event={"ID":"33277503-645e-402a-9629-c689c9a2df1f","Type":"ContainerStarted","Data":"483213fbdfe84466dd88fc960e29e96aef85efddf8f7414abd6e9582534ef8cf"}
Sep 09 05:51:14.810287 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:14.809944    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd8996fdc-chl5s" event={"ID":"33277503-645e-402a-9629-c689c9a2df1f","Type":"ContainerStarted","Data":"f2f544c65fd5ee9bb8d3fffce25ea1ee89f0d831c263bd81c72d397d8cae2b1d"}
Sep 09 05:51:14.830097 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:14.830052    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cd8996fdc-chl5s" podStartSLOduration=1.830037313 podStartE2EDuration="1.830037313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:51:14.8283271 +0000 UTC m=+102.044657013" watchObservedRunningTime="2026-09-09 05:51:14.830037313 +0000 UTC m=+102.046367198"
Sep 09 05:51:19.556339 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:19.556299    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:24.032616 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:24.032578    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:24.033121 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:24.032854    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:24.037253 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:24.037230    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:24.860704 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:24.860669    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:51:24.903658 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:24.903611    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-559c47d9b5-rrcbc"]
Sep 09 05:51:26.828557 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:26.828452    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv" podUID="48cf9622-0adf-4129-955f-3e82431ab4ab" containerName="registry" containerID="cri-o://87571929bfc050e373f6cc1a0cb96d715c1b4185e647406e2735f55d1f4fd9c3" gracePeriod=30
Sep 09 05:51:27.064451 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.064428    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:51:27.171825 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.171736    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls\") pod \"48cf9622-0adf-4129-955f-3e82431ab4ab\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") "
Sep 09 05:51:27.171825 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.171774    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4xrn\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-kube-api-access-c4xrn\") pod \"48cf9622-0adf-4129-955f-3e82431ab4ab\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") "
Sep 09 05:51:27.171825 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.171806    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-certificates\") pod \"48cf9622-0adf-4129-955f-3e82431ab4ab\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") "
Sep 09 05:51:27.172042 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.171838    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/48cf9622-0adf-4129-955f-3e82431ab4ab-image-registry-private-configuration\") pod \"48cf9622-0adf-4129-955f-3e82431ab4ab\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") "
Sep 09 05:51:27.172042 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.171977    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/48cf9622-0adf-4129-955f-3e82431ab4ab-ca-trust-extracted\") pod \"48cf9622-0adf-4129-955f-3e82431ab4ab\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") "
Sep 09 05:51:27.172140 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.172049    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/48cf9622-0adf-4129-955f-3e82431ab4ab-trusted-ca\") pod \"48cf9622-0adf-4129-955f-3e82431ab4ab\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") "
Sep 09 05:51:27.172140 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.172077    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/48cf9622-0adf-4129-955f-3e82431ab4ab-installation-pull-secrets\") pod \"48cf9622-0adf-4129-955f-3e82431ab4ab\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") "
Sep 09 05:51:27.172239 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.172161    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-bound-sa-token\") pod \"48cf9622-0adf-4129-955f-3e82431ab4ab\" (UID: \"48cf9622-0adf-4129-955f-3e82431ab4ab\") "
Sep 09 05:51:27.172554 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.172411    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "48cf9622-0adf-4129-955f-3e82431ab4ab" (UID: "48cf9622-0adf-4129-955f-3e82431ab4ab"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:51:27.172554 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.172513    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48cf9622-0adf-4129-955f-3e82431ab4ab-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "48cf9622-0adf-4129-955f-3e82431ab4ab" (UID: "48cf9622-0adf-4129-955f-3e82431ab4ab"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:51:27.174608 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.174564    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48cf9622-0adf-4129-955f-3e82431ab4ab-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "48cf9622-0adf-4129-955f-3e82431ab4ab" (UID: "48cf9622-0adf-4129-955f-3e82431ab4ab"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:51:27.174709 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.174611    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-kube-api-access-c4xrn" (OuterVolumeSpecName: "kube-api-access-c4xrn") pod "48cf9622-0adf-4129-955f-3e82431ab4ab" (UID: "48cf9622-0adf-4129-955f-3e82431ab4ab"). InnerVolumeSpecName "kube-api-access-c4xrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:51:27.174764 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.174708    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "48cf9622-0adf-4129-955f-3e82431ab4ab" (UID: "48cf9622-0adf-4129-955f-3e82431ab4ab"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:51:27.174851 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.174831    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "48cf9622-0adf-4129-955f-3e82431ab4ab" (UID: "48cf9622-0adf-4129-955f-3e82431ab4ab"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:51:27.175001 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.174976    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48cf9622-0adf-4129-955f-3e82431ab4ab-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "48cf9622-0adf-4129-955f-3e82431ab4ab" (UID: "48cf9622-0adf-4129-955f-3e82431ab4ab"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:51:27.180339 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.180318    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48cf9622-0adf-4129-955f-3e82431ab4ab-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "48cf9622-0adf-4129-955f-3e82431ab4ab" (UID: "48cf9622-0adf-4129-955f-3e82431ab4ab"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:51:27.273130 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.273087    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-bound-sa-token\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:27.273130 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.273123    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:27.273130 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.273135    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4xrn\" (UniqueName: \"kubernetes.io/projected/48cf9622-0adf-4129-955f-3e82431ab4ab-kube-api-access-c4xrn\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:27.273349 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.273147    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/48cf9622-0adf-4129-955f-3e82431ab4ab-registry-certificates\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:27.273349 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.273160    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/48cf9622-0adf-4129-955f-3e82431ab4ab-image-registry-private-configuration\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:27.273349 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.273172    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/48cf9622-0adf-4129-955f-3e82431ab4ab-ca-trust-extracted\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:27.273349 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.273183    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/48cf9622-0adf-4129-955f-3e82431ab4ab-trusted-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:27.273349 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.273194    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/48cf9622-0adf-4129-955f-3e82431ab4ab-installation-pull-secrets\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:27.870553 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.870516    2564 generic.go:356] "Generic (PLEG): container finished" podID="48cf9622-0adf-4129-955f-3e82431ab4ab" containerID="87571929bfc050e373f6cc1a0cb96d715c1b4185e647406e2735f55d1f4fd9c3" exitCode=0
Sep 09 05:51:27.870553 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.870556    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv"
Sep 09 05:51:27.871021 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.870591    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv" event={"ID":"48cf9622-0adf-4129-955f-3e82431ab4ab","Type":"ContainerDied","Data":"87571929bfc050e373f6cc1a0cb96d715c1b4185e647406e2735f55d1f4fd9c3"}
Sep 09 05:51:27.871021 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.870642    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64cdf68b7d-mdwmv" event={"ID":"48cf9622-0adf-4129-955f-3e82431ab4ab","Type":"ContainerDied","Data":"0f77db5c9a038493e49555bbf1d41adac7946c4cd21fd45d96fff11daf51be99"}
Sep 09 05:51:27.871021 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.870663    2564 scope.go:117] "RemoveContainer" containerID="87571929bfc050e373f6cc1a0cb96d715c1b4185e647406e2735f55d1f4fd9c3"
Sep 09 05:51:27.881094 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.881071    2564 scope.go:117] "RemoveContainer" containerID="87571929bfc050e373f6cc1a0cb96d715c1b4185e647406e2735f55d1f4fd9c3"
Sep 09 05:51:27.881415 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:51:27.881385    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87571929bfc050e373f6cc1a0cb96d715c1b4185e647406e2735f55d1f4fd9c3\": container with ID starting with 87571929bfc050e373f6cc1a0cb96d715c1b4185e647406e2735f55d1f4fd9c3 not found: ID does not exist" containerID="87571929bfc050e373f6cc1a0cb96d715c1b4185e647406e2735f55d1f4fd9c3"
Sep 09 05:51:27.881482 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.881417    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87571929bfc050e373f6cc1a0cb96d715c1b4185e647406e2735f55d1f4fd9c3"} err="failed to get container status \"87571929bfc050e373f6cc1a0cb96d715c1b4185e647406e2735f55d1f4fd9c3\": rpc error: code = NotFound desc = could not find container \"87571929bfc050e373f6cc1a0cb96d715c1b4185e647406e2735f55d1f4fd9c3\": container with ID starting with 87571929bfc050e373f6cc1a0cb96d715c1b4185e647406e2735f55d1f4fd9c3 not found: ID does not exist"
Sep 09 05:51:27.894223 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.894192    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64cdf68b7d-mdwmv"]
Sep 09 05:51:27.897549 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:27.897530    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64cdf68b7d-mdwmv"]
Sep 09 05:51:28.803780 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:28.803735    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5979cf6c99-xlfkv" podUID="72b883f9-df2b-4c15-ab55-79a6b927cd97" containerName="console" containerID="cri-o://a4f1c58d5425a9294f1d465a1142173ea04d9dd81015dd45cc1900f93308ff1f" gracePeriod=15
Sep 09 05:51:29.047454 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.047427    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:51:29.195358 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.195312    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-service-ca\") pod \"72b883f9-df2b-4c15-ab55-79a6b927cd97\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") "
Sep 09 05:51:29.195575 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.195374    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-serving-cert\") pod \"72b883f9-df2b-4c15-ab55-79a6b927cd97\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") "
Sep 09 05:51:29.195575 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.195428    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdspw\" (UniqueName: \"kubernetes.io/projected/72b883f9-df2b-4c15-ab55-79a6b927cd97-kube-api-access-cdspw\") pod \"72b883f9-df2b-4c15-ab55-79a6b927cd97\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") "
Sep 09 05:51:29.195575 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.195443    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-config\") pod \"72b883f9-df2b-4c15-ab55-79a6b927cd97\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") "
Sep 09 05:51:29.195575 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.195509    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-oauth-config\") pod \"72b883f9-df2b-4c15-ab55-79a6b927cd97\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") "
Sep 09 05:51:29.195575 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.195546    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-oauth-serving-cert\") pod \"72b883f9-df2b-4c15-ab55-79a6b927cd97\" (UID: \"72b883f9-df2b-4c15-ab55-79a6b927cd97\") "
Sep 09 05:51:29.195907 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.195874    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-service-ca" (OuterVolumeSpecName: "service-ca") pod "72b883f9-df2b-4c15-ab55-79a6b927cd97" (UID: "72b883f9-df2b-4c15-ab55-79a6b927cd97"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:51:29.195964 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.195916    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-config" (OuterVolumeSpecName: "console-config") pod "72b883f9-df2b-4c15-ab55-79a6b927cd97" (UID: "72b883f9-df2b-4c15-ab55-79a6b927cd97"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:51:29.196003 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.195963    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "72b883f9-df2b-4c15-ab55-79a6b927cd97" (UID: "72b883f9-df2b-4c15-ab55-79a6b927cd97"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:51:29.197895 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.197866    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b883f9-df2b-4c15-ab55-79a6b927cd97-kube-api-access-cdspw" (OuterVolumeSpecName: "kube-api-access-cdspw") pod "72b883f9-df2b-4c15-ab55-79a6b927cd97" (UID: "72b883f9-df2b-4c15-ab55-79a6b927cd97"). InnerVolumeSpecName "kube-api-access-cdspw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:51:29.197895 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.197887    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "72b883f9-df2b-4c15-ab55-79a6b927cd97" (UID: "72b883f9-df2b-4c15-ab55-79a6b927cd97"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:51:29.198038 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.197921    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "72b883f9-df2b-4c15-ab55-79a6b927cd97" (UID: "72b883f9-df2b-4c15-ab55-79a6b927cd97"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:51:29.296557 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.296523    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-oauth-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:29.296557 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.296553    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:29.296557 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.296563    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:29.296772 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.296572    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdspw\" (UniqueName: \"kubernetes.io/projected/72b883f9-df2b-4c15-ab55-79a6b927cd97-kube-api-access-cdspw\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:29.296772 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.296582    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:29.296772 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.296590    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72b883f9-df2b-4c15-ab55-79a6b927cd97-console-oauth-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:29.301572 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.301542    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48cf9622-0adf-4129-955f-3e82431ab4ab" path="/var/lib/kubelet/pods/48cf9622-0adf-4129-955f-3e82431ab4ab/volumes"
Sep 09 05:51:29.880915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.880881    2564 generic.go:356] "Generic (PLEG): container finished" podID="72b883f9-df2b-4c15-ab55-79a6b927cd97" containerID="a4f1c58d5425a9294f1d465a1142173ea04d9dd81015dd45cc1900f93308ff1f" exitCode=0
Sep 09 05:51:29.881069 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.880921    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5979cf6c99-xlfkv"
Sep 09 05:51:29.881069 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.880944    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5979cf6c99-xlfkv" event={"ID":"72b883f9-df2b-4c15-ab55-79a6b927cd97","Type":"ContainerDied","Data":"a4f1c58d5425a9294f1d465a1142173ea04d9dd81015dd45cc1900f93308ff1f"}
Sep 09 05:51:29.881069 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.880980    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5979cf6c99-xlfkv" event={"ID":"72b883f9-df2b-4c15-ab55-79a6b927cd97","Type":"ContainerDied","Data":"5ef73391f3ab84d76a1cf425f2a607f3f868a82270f9190e9cd5b8777f3dd693"}
Sep 09 05:51:29.881069 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.880999    2564 scope.go:117] "RemoveContainer" containerID="a4f1c58d5425a9294f1d465a1142173ea04d9dd81015dd45cc1900f93308ff1f"
Sep 09 05:51:29.889157 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.889093    2564 scope.go:117] "RemoveContainer" containerID="a4f1c58d5425a9294f1d465a1142173ea04d9dd81015dd45cc1900f93308ff1f"
Sep 09 05:51:29.889390 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:51:29.889370    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f1c58d5425a9294f1d465a1142173ea04d9dd81015dd45cc1900f93308ff1f\": container with ID starting with a4f1c58d5425a9294f1d465a1142173ea04d9dd81015dd45cc1900f93308ff1f not found: ID does not exist" containerID="a4f1c58d5425a9294f1d465a1142173ea04d9dd81015dd45cc1900f93308ff1f"
Sep 09 05:51:29.889479 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.889398    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f1c58d5425a9294f1d465a1142173ea04d9dd81015dd45cc1900f93308ff1f"} err="failed to get container status \"a4f1c58d5425a9294f1d465a1142173ea04d9dd81015dd45cc1900f93308ff1f\": rpc error: code = NotFound desc = could not find container \"a4f1c58d5425a9294f1d465a1142173ea04d9dd81015dd45cc1900f93308ff1f\": container with ID starting with a4f1c58d5425a9294f1d465a1142173ea04d9dd81015dd45cc1900f93308ff1f not found: ID does not exist"
Sep 09 05:51:29.897555 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.897529    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5979cf6c99-xlfkv"]
Sep 09 05:51:29.899465 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:29.899443    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5979cf6c99-xlfkv"]
Sep 09 05:51:31.301723 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:31.301693    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72b883f9-df2b-4c15-ab55-79a6b927cd97" path="/var/lib/kubelet/pods/72b883f9-df2b-4c15-ab55-79a6b927cd97/volumes"
Sep 09 05:51:37.914878 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:37.914847    2564 generic.go:356] "Generic (PLEG): container finished" podID="6cd6c398-b2b7-4c95-8899-18f779265279" containerID="0530d4c5697a48ea53dc89f625a0b3291295d251310fad0ec18f100581ec13fe" exitCode=0
Sep 09 05:51:37.915299 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:37.914924    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm" event={"ID":"6cd6c398-b2b7-4c95-8899-18f779265279","Type":"ContainerDied","Data":"0530d4c5697a48ea53dc89f625a0b3291295d251310fad0ec18f100581ec13fe"}
Sep 09 05:51:37.915363 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:37.915300    2564 scope.go:117] "RemoveContainer" containerID="0530d4c5697a48ea53dc89f625a0b3291295d251310fad0ec18f100581ec13fe"
Sep 09 05:51:37.916546 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:37.916523    2564 generic.go:356] "Generic (PLEG): container finished" podID="d1b76aac-f159-4940-8a85-64a227c18b05" containerID="59eb5f42f4334dafadefe210edf8b975048ff08744534cf1dff8a7484feb0bb8" exitCode=0
Sep 09 05:51:37.916632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:37.916574    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz" event={"ID":"d1b76aac-f159-4940-8a85-64a227c18b05","Type":"ContainerDied","Data":"59eb5f42f4334dafadefe210edf8b975048ff08744534cf1dff8a7484feb0bb8"}
Sep 09 05:51:37.916910 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:37.916894    2564 scope.go:117] "RemoveContainer" containerID="59eb5f42f4334dafadefe210edf8b975048ff08744534cf1dff8a7484feb0bb8"
Sep 09 05:51:38.697771 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:38.697702    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5dc9546fb7-s84mv" podUID="d1ff1aaa-b886-434e-8c4a-06f73f033576" containerName="console" containerID="cri-o://e7175a0f68898300af7402bf35e154944a84be742e3c7456201f013eabb26b23" gracePeriod=15
Sep 09 05:51:38.923147 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:38.923116    2564 generic.go:356] "Generic (PLEG): container finished" podID="d1ff1aaa-b886-434e-8c4a-06f73f033576" containerID="e7175a0f68898300af7402bf35e154944a84be742e3c7456201f013eabb26b23" exitCode=0
Sep 09 05:51:38.923558 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:38.923185    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dc9546fb7-s84mv" event={"ID":"d1ff1aaa-b886-434e-8c4a-06f73f033576","Type":"ContainerDied","Data":"e7175a0f68898300af7402bf35e154944a84be742e3c7456201f013eabb26b23"}
Sep 09 05:51:38.925156 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:38.925132    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-cflnz" event={"ID":"d1b76aac-f159-4940-8a85-64a227c18b05","Type":"ContainerStarted","Data":"3127030b2b5fa1ea61691dcc54196d4b1a44719c9ebcb001aad6c91863c71e18"}
Sep 09 05:51:38.927100 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:38.927081    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8hmqm" event={"ID":"6cd6c398-b2b7-4c95-8899-18f779265279","Type":"ContainerStarted","Data":"149cd6db668a5fa452d9b2249c409a67e40ded9bd0f8f83cc5ae4b25d42adf1a"}
Sep 09 05:51:38.957156 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:38.957080    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:39.077833 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.077800    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-oauth-serving-cert\") pod \"d1ff1aaa-b886-434e-8c4a-06f73f033576\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") "
Sep 09 05:51:39.078019 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.077848    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-config\") pod \"d1ff1aaa-b886-434e-8c4a-06f73f033576\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") "
Sep 09 05:51:39.078019 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.077868    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-service-ca\") pod \"d1ff1aaa-b886-434e-8c4a-06f73f033576\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") "
Sep 09 05:51:39.078019 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.077919    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-serving-cert\") pod \"d1ff1aaa-b886-434e-8c4a-06f73f033576\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") "
Sep 09 05:51:39.078019 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.077959    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm74k\" (UniqueName: \"kubernetes.io/projected/d1ff1aaa-b886-434e-8c4a-06f73f033576-kube-api-access-mm74k\") pod \"d1ff1aaa-b886-434e-8c4a-06f73f033576\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") "
Sep 09 05:51:39.078019 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.077992    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-trusted-ca-bundle\") pod \"d1ff1aaa-b886-434e-8c4a-06f73f033576\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") "
Sep 09 05:51:39.078273 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.078021    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-oauth-config\") pod \"d1ff1aaa-b886-434e-8c4a-06f73f033576\" (UID: \"d1ff1aaa-b886-434e-8c4a-06f73f033576\") "
Sep 09 05:51:39.078273 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.078221    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d1ff1aaa-b886-434e-8c4a-06f73f033576" (UID: "d1ff1aaa-b886-434e-8c4a-06f73f033576"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:51:39.078273 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.078252    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-service-ca" (OuterVolumeSpecName: "service-ca") pod "d1ff1aaa-b886-434e-8c4a-06f73f033576" (UID: "d1ff1aaa-b886-434e-8c4a-06f73f033576"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:51:39.078409 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.078318    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-config" (OuterVolumeSpecName: "console-config") pod "d1ff1aaa-b886-434e-8c4a-06f73f033576" (UID: "d1ff1aaa-b886-434e-8c4a-06f73f033576"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:51:39.078409 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.078385    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d1ff1aaa-b886-434e-8c4a-06f73f033576" (UID: "d1ff1aaa-b886-434e-8c4a-06f73f033576"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:51:39.078967 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.078944    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:39.079150 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.078970    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:39.079150 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.078986    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-trusted-ca-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:39.079150 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.078999    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1ff1aaa-b886-434e-8c4a-06f73f033576-oauth-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:39.080254 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.080227    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d1ff1aaa-b886-434e-8c4a-06f73f033576" (UID: "d1ff1aaa-b886-434e-8c4a-06f73f033576"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:51:39.080338 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.080274    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ff1aaa-b886-434e-8c4a-06f73f033576-kube-api-access-mm74k" (OuterVolumeSpecName: "kube-api-access-mm74k") pod "d1ff1aaa-b886-434e-8c4a-06f73f033576" (UID: "d1ff1aaa-b886-434e-8c4a-06f73f033576"). InnerVolumeSpecName "kube-api-access-mm74k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:51:39.080338 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.080310    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d1ff1aaa-b886-434e-8c4a-06f73f033576" (UID: "d1ff1aaa-b886-434e-8c4a-06f73f033576"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:51:39.180072 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.180041    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:39.180072 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.180068    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm74k\" (UniqueName: \"kubernetes.io/projected/d1ff1aaa-b886-434e-8c4a-06f73f033576-kube-api-access-mm74k\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:39.180072 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.180077    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1ff1aaa-b886-434e-8c4a-06f73f033576-console-oauth-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:39.932517 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.932464    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dc9546fb7-s84mv"
Sep 09 05:51:39.932987 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.932469    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dc9546fb7-s84mv" event={"ID":"d1ff1aaa-b886-434e-8c4a-06f73f033576","Type":"ContainerDied","Data":"c73cf8b49127ae3a016fb10564d469bcfed2202725dffaf2893fd8b35c5201da"}
Sep 09 05:51:39.932987 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.932619    2564 scope.go:117] "RemoveContainer" containerID="e7175a0f68898300af7402bf35e154944a84be742e3c7456201f013eabb26b23"
Sep 09 05:51:39.949134 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.949110    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dc9546fb7-s84mv"]
Sep 09 05:51:39.952969 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:39.952949    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dc9546fb7-s84mv"]
Sep 09 05:51:41.300937 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:41.300901    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1ff1aaa-b886-434e-8c4a-06f73f033576" path="/var/lib/kubelet/pods/d1ff1aaa-b886-434e-8c4a-06f73f033576/volumes"
Sep 09 05:51:47.966331 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:47.966252    2564 generic.go:356] "Generic (PLEG): container finished" podID="c2ce177a-74d8-4240-8eb1-bf8c703e59d6" containerID="d98f2a9cd9cb73fc8b5fd73c3cbeac9fc836889a5405f6489e93331f0d82a168" exitCode=0
Sep 09 05:51:47.966683 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:47.966327    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp" event={"ID":"c2ce177a-74d8-4240-8eb1-bf8c703e59d6","Type":"ContainerDied","Data":"d98f2a9cd9cb73fc8b5fd73c3cbeac9fc836889a5405f6489e93331f0d82a168"}
Sep 09 05:51:47.966683 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:47.966671    2564 scope.go:117] "RemoveContainer" containerID="d98f2a9cd9cb73fc8b5fd73c3cbeac9fc836889a5405f6489e93331f0d82a168"
Sep 09 05:51:48.973113 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:48.973082    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rghsp" event={"ID":"c2ce177a-74d8-4240-8eb1-bf8c703e59d6","Type":"ContainerStarted","Data":"89378a10697b94719c8c39cd12634fad9e7e40a0d154f7dfc7f43ea45021ba18"}
Sep 09 05:51:49.926228 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:49.926174    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-559c47d9b5-rrcbc" podUID="845be81c-3808-4bd3-90bb-fc9b3a4a198c" containerName="console" containerID="cri-o://75dbb3f49cd0654305f5467f1b60fbc1300a127ae5edec3ad0a6bbac69cfeb7a" gracePeriod=15
Sep 09 05:51:50.187914 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.187851    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:51:50.362013 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.361982    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-service-ca\") pod \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") "
Sep 09 05:51:50.362171 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.362032    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-serving-cert\") pod \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") "
Sep 09 05:51:50.362171 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.362055    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwjht\" (UniqueName: \"kubernetes.io/projected/845be81c-3808-4bd3-90bb-fc9b3a4a198c-kube-api-access-xwjht\") pod \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") "
Sep 09 05:51:50.362171 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.362136    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-config\") pod \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") "
Sep 09 05:51:50.362351 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.362194    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-oauth-config\") pod \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") "
Sep 09 05:51:50.362351 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.362224    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-oauth-serving-cert\") pod \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") "
Sep 09 05:51:50.362351 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.362281    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-trusted-ca-bundle\") pod \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\" (UID: \"845be81c-3808-4bd3-90bb-fc9b3a4a198c\") "
Sep 09 05:51:50.362487 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.362365    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-service-ca" (OuterVolumeSpecName: "service-ca") pod "845be81c-3808-4bd3-90bb-fc9b3a4a198c" (UID: "845be81c-3808-4bd3-90bb-fc9b3a4a198c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:51:50.362487 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.362458    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-config" (OuterVolumeSpecName: "console-config") pod "845be81c-3808-4bd3-90bb-fc9b3a4a198c" (UID: "845be81c-3808-4bd3-90bb-fc9b3a4a198c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:51:50.362623 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.362596    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:50.362623 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.362615    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:50.362722 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.362695    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "845be81c-3808-4bd3-90bb-fc9b3a4a198c" (UID: "845be81c-3808-4bd3-90bb-fc9b3a4a198c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:51:50.362774 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.362743    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "845be81c-3808-4bd3-90bb-fc9b3a4a198c" (UID: "845be81c-3808-4bd3-90bb-fc9b3a4a198c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:51:50.364346 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.364301    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "845be81c-3808-4bd3-90bb-fc9b3a4a198c" (UID: "845be81c-3808-4bd3-90bb-fc9b3a4a198c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:51:50.364346 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.364319    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845be81c-3808-4bd3-90bb-fc9b3a4a198c-kube-api-access-xwjht" (OuterVolumeSpecName: "kube-api-access-xwjht") pod "845be81c-3808-4bd3-90bb-fc9b3a4a198c" (UID: "845be81c-3808-4bd3-90bb-fc9b3a4a198c"). InnerVolumeSpecName "kube-api-access-xwjht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:51:50.364522 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.364404    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "845be81c-3808-4bd3-90bb-fc9b3a4a198c" (UID: "845be81c-3808-4bd3-90bb-fc9b3a4a198c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:51:50.463222 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.463146    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:50.463222 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.463174    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwjht\" (UniqueName: \"kubernetes.io/projected/845be81c-3808-4bd3-90bb-fc9b3a4a198c-kube-api-access-xwjht\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:50.463222 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.463184    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/845be81c-3808-4bd3-90bb-fc9b3a4a198c-console-oauth-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:50.463222 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.463193    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-oauth-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:50.463222 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.463203    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/845be81c-3808-4bd3-90bb-fc9b3a4a198c-trusted-ca-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:51:50.984508 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.984458    2564 generic.go:356] "Generic (PLEG): container finished" podID="845be81c-3808-4bd3-90bb-fc9b3a4a198c" containerID="75dbb3f49cd0654305f5467f1b60fbc1300a127ae5edec3ad0a6bbac69cfeb7a" exitCode=0
Sep 09 05:51:50.984668 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.984520    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559c47d9b5-rrcbc"
Sep 09 05:51:50.984668 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.984521    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559c47d9b5-rrcbc" event={"ID":"845be81c-3808-4bd3-90bb-fc9b3a4a198c","Type":"ContainerDied","Data":"75dbb3f49cd0654305f5467f1b60fbc1300a127ae5edec3ad0a6bbac69cfeb7a"}
Sep 09 05:51:50.984668 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.984640    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559c47d9b5-rrcbc" event={"ID":"845be81c-3808-4bd3-90bb-fc9b3a4a198c","Type":"ContainerDied","Data":"549cac575b8fba1748bec4362fa3bfbfec843dff73b81d0e4b3d4b826a328980"}
Sep 09 05:51:50.984771 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.984669    2564 scope.go:117] "RemoveContainer" containerID="75dbb3f49cd0654305f5467f1b60fbc1300a127ae5edec3ad0a6bbac69cfeb7a"
Sep 09 05:51:50.999397 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.999370    2564 scope.go:117] "RemoveContainer" containerID="75dbb3f49cd0654305f5467f1b60fbc1300a127ae5edec3ad0a6bbac69cfeb7a"
Sep 09 05:51:50.999898 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:51:50.999866    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75dbb3f49cd0654305f5467f1b60fbc1300a127ae5edec3ad0a6bbac69cfeb7a\": container with ID starting with 75dbb3f49cd0654305f5467f1b60fbc1300a127ae5edec3ad0a6bbac69cfeb7a not found: ID does not exist" containerID="75dbb3f49cd0654305f5467f1b60fbc1300a127ae5edec3ad0a6bbac69cfeb7a"
Sep 09 05:51:51.000003 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:50.999911    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75dbb3f49cd0654305f5467f1b60fbc1300a127ae5edec3ad0a6bbac69cfeb7a"} err="failed to get container status \"75dbb3f49cd0654305f5467f1b60fbc1300a127ae5edec3ad0a6bbac69cfeb7a\": rpc error: code = NotFound desc = could not find container \"75dbb3f49cd0654305f5467f1b60fbc1300a127ae5edec3ad0a6bbac69cfeb7a\": container with ID starting with 75dbb3f49cd0654305f5467f1b60fbc1300a127ae5edec3ad0a6bbac69cfeb7a not found: ID does not exist"
Sep 09 05:51:51.012992 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:51.012969    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-559c47d9b5-rrcbc"]
Sep 09 05:51:51.016592 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:51.016571    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-559c47d9b5-rrcbc"]
Sep 09 05:51:51.302438 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:51:51.302365    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="845be81c-3808-4bd3-90bb-fc9b3a4a198c" path="/var/lib/kubelet/pods/845be81c-3808-4bd3-90bb-fc9b3a4a198c/volumes"
Sep 09 05:52:24.825392 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:24.825355    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 05:52:24.825884 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:24.825664    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="alertmanager" containerID="cri-o://ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0" gracePeriod=120
Sep 09 05:52:24.825884 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:24.825715    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="kube-rbac-proxy-metric" containerID="cri-o://4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90" gracePeriod=120
Sep 09 05:52:24.825884 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:24.825747    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="kube-rbac-proxy-web" containerID="cri-o://f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e" gracePeriod=120
Sep 09 05:52:24.825884 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:24.825769    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="prom-label-proxy" containerID="cri-o://a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb" gracePeriod=120
Sep 09 05:52:24.825884 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:24.825798    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="config-reloader" containerID="cri-o://0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2" gracePeriod=120
Sep 09 05:52:24.826159 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:24.825798    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="kube-rbac-proxy" containerID="cri-o://f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d" gracePeriod=120
Sep 09 05:52:25.125618 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:25.125521    2564 generic.go:356] "Generic (PLEG): container finished" podID="5d98f6c8-6c49-44df-8610-2583c98be150" containerID="a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb" exitCode=0
Sep 09 05:52:25.125618 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:25.125549    2564 generic.go:356] "Generic (PLEG): container finished" podID="5d98f6c8-6c49-44df-8610-2583c98be150" containerID="f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d" exitCode=0
Sep 09 05:52:25.125618 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:25.125558    2564 generic.go:356] "Generic (PLEG): container finished" podID="5d98f6c8-6c49-44df-8610-2583c98be150" containerID="0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2" exitCode=0
Sep 09 05:52:25.125618 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:25.125568    2564 generic.go:356] "Generic (PLEG): container finished" podID="5d98f6c8-6c49-44df-8610-2583c98be150" containerID="ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0" exitCode=0
Sep 09 05:52:25.125618 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:25.125597    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5d98f6c8-6c49-44df-8610-2583c98be150","Type":"ContainerDied","Data":"a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb"}
Sep 09 05:52:25.125875 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:25.125669    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5d98f6c8-6c49-44df-8610-2583c98be150","Type":"ContainerDied","Data":"f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d"}
Sep 09 05:52:25.125875 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:25.125699    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5d98f6c8-6c49-44df-8610-2583c98be150","Type":"ContainerDied","Data":"0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2"}
Sep 09 05:52:25.125875 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:25.125724    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5d98f6c8-6c49-44df-8610-2583c98be150","Type":"ContainerDied","Data":"ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0"}
Sep 09 05:52:26.069290 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.069267    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.127339 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.127311    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-cluster-tls-config\") pod \"5d98f6c8-6c49-44df-8610-2583c98be150\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") "
Sep 09 05:52:26.127538 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.127351    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5d98f6c8-6c49-44df-8610-2583c98be150-config-out\") pod \"5d98f6c8-6c49-44df-8610-2583c98be150\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") "
Sep 09 05:52:26.127538 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.127379    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d98f6c8-6c49-44df-8610-2583c98be150-metrics-client-ca\") pod \"5d98f6c8-6c49-44df-8610-2583c98be150\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") "
Sep 09 05:52:26.127538 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.127421    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy-web\") pod \"5d98f6c8-6c49-44df-8610-2583c98be150\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") "
Sep 09 05:52:26.127538 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.127456    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5d98f6c8-6c49-44df-8610-2583c98be150-tls-assets\") pod \"5d98f6c8-6c49-44df-8610-2583c98be150\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") "
Sep 09 05:52:26.127538 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.127504    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-config-volume\") pod \"5d98f6c8-6c49-44df-8610-2583c98be150\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") "
Sep 09 05:52:26.127538 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.127533    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d98f6c8-6c49-44df-8610-2583c98be150-alertmanager-trusted-ca-bundle\") pod \"5d98f6c8-6c49-44df-8610-2583c98be150\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") "
Sep 09 05:52:26.127834 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.127561    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnbxx\" (UniqueName: \"kubernetes.io/projected/5d98f6c8-6c49-44df-8610-2583c98be150-kube-api-access-fnbxx\") pod \"5d98f6c8-6c49-44df-8610-2583c98be150\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") "
Sep 09 05:52:26.127834 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.127590    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5d98f6c8-6c49-44df-8610-2583c98be150\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") "
Sep 09 05:52:26.127834 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.127615    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5d98f6c8-6c49-44df-8610-2583c98be150-alertmanager-main-db\") pod \"5d98f6c8-6c49-44df-8610-2583c98be150\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") "
Sep 09 05:52:26.127834 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.127644    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy\") pod \"5d98f6c8-6c49-44df-8610-2583c98be150\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") "
Sep 09 05:52:26.128044 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.128010    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d98f6c8-6c49-44df-8610-2583c98be150-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5d98f6c8-6c49-44df-8610-2583c98be150" (UID: "5d98f6c8-6c49-44df-8610-2583c98be150"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:52:26.128118 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.128080    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d98f6c8-6c49-44df-8610-2583c98be150-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5d98f6c8-6c49-44df-8610-2583c98be150" (UID: "5d98f6c8-6c49-44df-8610-2583c98be150"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:52:26.128118 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.128101    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-main-tls\") pod \"5d98f6c8-6c49-44df-8610-2583c98be150\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") "
Sep 09 05:52:26.128215 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.128189    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-web-config\") pod \"5d98f6c8-6c49-44df-8610-2583c98be150\" (UID: \"5d98f6c8-6c49-44df-8610-2583c98be150\") "
Sep 09 05:52:26.128546 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.128520    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d98f6c8-6c49-44df-8610-2583c98be150-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5d98f6c8-6c49-44df-8610-2583c98be150" (UID: "5d98f6c8-6c49-44df-8610-2583c98be150"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:52:26.128657 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.128576    2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d98f6c8-6c49-44df-8610-2583c98be150-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:52:26.128657 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.128595    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d98f6c8-6c49-44df-8610-2583c98be150-metrics-client-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:52:26.131368 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.131339    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d98f6c8-6c49-44df-8610-2583c98be150-kube-api-access-fnbxx" (OuterVolumeSpecName: "kube-api-access-fnbxx") pod "5d98f6c8-6c49-44df-8610-2583c98be150" (UID: "5d98f6c8-6c49-44df-8610-2583c98be150"). InnerVolumeSpecName "kube-api-access-fnbxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:52:26.131502 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.131363    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5d98f6c8-6c49-44df-8610-2583c98be150" (UID: "5d98f6c8-6c49-44df-8610-2583c98be150"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:52:26.132104 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.132064    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-config-volume" (OuterVolumeSpecName: "config-volume") pod "5d98f6c8-6c49-44df-8610-2583c98be150" (UID: "5d98f6c8-6c49-44df-8610-2583c98be150"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:52:26.132405 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.132378    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d98f6c8-6c49-44df-8610-2583c98be150-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5d98f6c8-6c49-44df-8610-2583c98be150" (UID: "5d98f6c8-6c49-44df-8610-2583c98be150"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:52:26.132784 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.132759    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5d98f6c8-6c49-44df-8610-2583c98be150" (UID: "5d98f6c8-6c49-44df-8610-2583c98be150"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:52:26.133183 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.133158    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d98f6c8-6c49-44df-8610-2583c98be150-config-out" (OuterVolumeSpecName: "config-out") pod "5d98f6c8-6c49-44df-8610-2583c98be150" (UID: "5d98f6c8-6c49-44df-8610-2583c98be150"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:52:26.133279 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.133228    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5d98f6c8-6c49-44df-8610-2583c98be150" (UID: "5d98f6c8-6c49-44df-8610-2583c98be150"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:52:26.133365 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.133340    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5d98f6c8-6c49-44df-8610-2583c98be150" (UID: "5d98f6c8-6c49-44df-8610-2583c98be150"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:52:26.136448 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.136410    2564 generic.go:356] "Generic (PLEG): container finished" podID="5d98f6c8-6c49-44df-8610-2583c98be150" containerID="4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90" exitCode=0
Sep 09 05:52:26.136448 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.136438    2564 generic.go:356] "Generic (PLEG): container finished" podID="5d98f6c8-6c49-44df-8610-2583c98be150" containerID="f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e" exitCode=0
Sep 09 05:52:26.136616 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.136598    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.136670 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.136623    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5d98f6c8-6c49-44df-8610-2583c98be150","Type":"ContainerDied","Data":"4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90"}
Sep 09 05:52:26.136729 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.136703    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5d98f6c8-6c49-44df-8610-2583c98be150","Type":"ContainerDied","Data":"f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e"}
Sep 09 05:52:26.136783 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.136750    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5d98f6c8-6c49-44df-8610-2583c98be150","Type":"ContainerDied","Data":"ed5a5cf04fcc4785208885d6d923377cb606e3bf0727ee4e7039718b8daf1b1f"}
Sep 09 05:52:26.136833 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.136800    2564 scope.go:117] "RemoveContainer" containerID="a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb"
Sep 09 05:52:26.137258 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.137234    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5d98f6c8-6c49-44df-8610-2583c98be150" (UID: "5d98f6c8-6c49-44df-8610-2583c98be150"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:52:26.144208 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.144183    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-web-config" (OuterVolumeSpecName: "web-config") pod "5d98f6c8-6c49-44df-8610-2583c98be150" (UID: "5d98f6c8-6c49-44df-8610-2583c98be150"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:52:26.149056 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.149039    2564 scope.go:117] "RemoveContainer" containerID="4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90"
Sep 09 05:52:26.156325 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.156304    2564 scope.go:117] "RemoveContainer" containerID="f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d"
Sep 09 05:52:26.163275 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.163253    2564 scope.go:117] "RemoveContainer" containerID="f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e"
Sep 09 05:52:26.169752 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.169737    2564 scope.go:117] "RemoveContainer" containerID="0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2"
Sep 09 05:52:26.176638 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.176619    2564 scope.go:117] "RemoveContainer" containerID="ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0"
Sep 09 05:52:26.183481 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.183464    2564 scope.go:117] "RemoveContainer" containerID="9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454"
Sep 09 05:52:26.190117 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.190100    2564 scope.go:117] "RemoveContainer" containerID="a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb"
Sep 09 05:52:26.190372 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:52:26.190354    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb\": container with ID starting with a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb not found: ID does not exist" containerID="a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb"
Sep 09 05:52:26.190429 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.190381    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb"} err="failed to get container status \"a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb\": rpc error: code = NotFound desc = could not find container \"a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb\": container with ID starting with a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb not found: ID does not exist"
Sep 09 05:52:26.190429 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.190399    2564 scope.go:117] "RemoveContainer" containerID="4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90"
Sep 09 05:52:26.190655 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:52:26.190637    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90\": container with ID starting with 4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90 not found: ID does not exist" containerID="4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90"
Sep 09 05:52:26.190707 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.190662    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90"} err="failed to get container status \"4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90\": rpc error: code = NotFound desc = could not find container \"4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90\": container with ID starting with 4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90 not found: ID does not exist"
Sep 09 05:52:26.190707 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.190683    2564 scope.go:117] "RemoveContainer" containerID="f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d"
Sep 09 05:52:26.190933 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:52:26.190916    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d\": container with ID starting with f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d not found: ID does not exist" containerID="f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d"
Sep 09 05:52:26.191001 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.190940    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d"} err="failed to get container status \"f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d\": rpc error: code = NotFound desc = could not find container \"f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d\": container with ID starting with f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d not found: ID does not exist"
Sep 09 05:52:26.191001 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.190961    2564 scope.go:117] "RemoveContainer" containerID="f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e"
Sep 09 05:52:26.191203 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:52:26.191186    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e\": container with ID starting with f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e not found: ID does not exist" containerID="f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e"
Sep 09 05:52:26.191240 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.191208    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e"} err="failed to get container status \"f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e\": rpc error: code = NotFound desc = could not find container \"f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e\": container with ID starting with f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e not found: ID does not exist"
Sep 09 05:52:26.191240 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.191222    2564 scope.go:117] "RemoveContainer" containerID="0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2"
Sep 09 05:52:26.191422 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:52:26.191408    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2\": container with ID starting with 0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2 not found: ID does not exist" containerID="0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2"
Sep 09 05:52:26.191467 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.191428    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2"} err="failed to get container status \"0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2\": rpc error: code = NotFound desc = could not find container \"0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2\": container with ID starting with 0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2 not found: ID does not exist"
Sep 09 05:52:26.191467 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.191441    2564 scope.go:117] "RemoveContainer" containerID="ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0"
Sep 09 05:52:26.191667 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:52:26.191650    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0\": container with ID starting with ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0 not found: ID does not exist" containerID="ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0"
Sep 09 05:52:26.191733 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.191673    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0"} err="failed to get container status \"ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0\": rpc error: code = NotFound desc = could not find container \"ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0\": container with ID starting with ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0 not found: ID does not exist"
Sep 09 05:52:26.191733 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.191692    2564 scope.go:117] "RemoveContainer" containerID="9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454"
Sep 09 05:52:26.191944 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:52:26.191926    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454\": container with ID starting with 9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454 not found: ID does not exist" containerID="9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454"
Sep 09 05:52:26.191982 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.191948    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454"} err="failed to get container status \"9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454\": rpc error: code = NotFound desc = could not find container \"9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454\": container with ID starting with 9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454 not found: ID does not exist"
Sep 09 05:52:26.191982 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.191965    2564 scope.go:117] "RemoveContainer" containerID="a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb"
Sep 09 05:52:26.192163 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.192146    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb"} err="failed to get container status \"a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb\": rpc error: code = NotFound desc = could not find container \"a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb\": container with ID starting with a618254f85511a4c35b129b5b88bfde7d2709c1a58cacce684f044ca4de18adb not found: ID does not exist"
Sep 09 05:52:26.192226 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.192164    2564 scope.go:117] "RemoveContainer" containerID="4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90"
Sep 09 05:52:26.192367 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.192348    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90"} err="failed to get container status \"4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90\": rpc error: code = NotFound desc = could not find container \"4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90\": container with ID starting with 4ecd7fe00d90445b9b9ce781ac48a12f1419786729309adc9fe85b00d509ef90 not found: ID does not exist"
Sep 09 05:52:26.192411 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.192368    2564 scope.go:117] "RemoveContainer" containerID="f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d"
Sep 09 05:52:26.192584 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.192567    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d"} err="failed to get container status \"f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d\": rpc error: code = NotFound desc = could not find container \"f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d\": container with ID starting with f9c285b98077c9e1cd5ef97d91667882ae36348c3cedbd94d0a8a82125380d3d not found: ID does not exist"
Sep 09 05:52:26.192658 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.192586    2564 scope.go:117] "RemoveContainer" containerID="f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e"
Sep 09 05:52:26.192825 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.192805    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e"} err="failed to get container status \"f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e\": rpc error: code = NotFound desc = could not find container \"f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e\": container with ID starting with f6f5db416af44a4b676f85cb559ba621f1d136fc7040c8176eebe24a51bac80e not found: ID does not exist"
Sep 09 05:52:26.192869 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.192826    2564 scope.go:117] "RemoveContainer" containerID="0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2"
Sep 09 05:52:26.193032 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.193014    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2"} err="failed to get container status \"0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2\": rpc error: code = NotFound desc = could not find container \"0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2\": container with ID starting with 0e56232e02f69880d18da0f686b6d82acb156e134ac53270a57ed1f1192a1cf2 not found: ID does not exist"
Sep 09 05:52:26.193098 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.193033    2564 scope.go:117] "RemoveContainer" containerID="ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0"
Sep 09 05:52:26.193239 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.193222    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0"} err="failed to get container status \"ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0\": rpc error: code = NotFound desc = could not find container \"ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0\": container with ID starting with ec9f5d8c8c62e18053d3716490f10bff706c2971bd57716a8fb7de3de89c1fa0 not found: ID does not exist"
Sep 09 05:52:26.193285 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.193238    2564 scope.go:117] "RemoveContainer" containerID="9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454"
Sep 09 05:52:26.193422 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.193404    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454"} err="failed to get container status \"9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454\": rpc error: code = NotFound desc = could not find container \"9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454\": container with ID starting with 9f3ae58d9dac895929daafae26c513bb9b167b302f2640ee79bf3e0caac6a454 not found: ID does not exist"
Sep 09 05:52:26.229970 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.229938    2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-web-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:52:26.229970 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.229967    2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-cluster-tls-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:52:26.229970 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.229976    2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5d98f6c8-6c49-44df-8610-2583c98be150-config-out\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:52:26.230154 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.229985    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:52:26.230154 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.229994    2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5d98f6c8-6c49-44df-8610-2583c98be150-tls-assets\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:52:26.230154 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.230004    2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-config-volume\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:52:26.230154 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.230012    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnbxx\" (UniqueName: \"kubernetes.io/projected/5d98f6c8-6c49-44df-8610-2583c98be150-kube-api-access-fnbxx\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:52:26.230154 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.230020    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:52:26.230154 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.230029    2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5d98f6c8-6c49-44df-8610-2583c98be150-alertmanager-main-db\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:52:26.230154 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.230037    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:52:26.230154 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.230046    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5d98f6c8-6c49-44df-8610-2583c98be150-secret-alertmanager-main-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:52:26.461168 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.461140    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 05:52:26.464790 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.464765    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 05:52:26.489380 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489351    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 05:52:26.489670 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489655    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b883f9-df2b-4c15-ab55-79a6b927cd97" containerName="console"
Sep 09 05:52:26.489712 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489671    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b883f9-df2b-4c15-ab55-79a6b927cd97" containerName="console"
Sep 09 05:52:26.489712 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489680    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48cf9622-0adf-4129-955f-3e82431ab4ab" containerName="registry"
Sep 09 05:52:26.489712 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489687    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="48cf9622-0adf-4129-955f-3e82431ab4ab" containerName="registry"
Sep 09 05:52:26.489712 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489693    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="kube-rbac-proxy-web"
Sep 09 05:52:26.489712 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489698    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="kube-rbac-proxy-web"
Sep 09 05:52:26.489712 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489704    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="845be81c-3808-4bd3-90bb-fc9b3a4a198c" containerName="console"
Sep 09 05:52:26.489712 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489709    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="845be81c-3808-4bd3-90bb-fc9b3a4a198c" containerName="console"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489716    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="alertmanager"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489721    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="alertmanager"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489729    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="init-config-reloader"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489734    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="init-config-reloader"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489745    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="prom-label-proxy"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489750    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="prom-label-proxy"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489756    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="kube-rbac-proxy-metric"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489761    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="kube-rbac-proxy-metric"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489769    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1ff1aaa-b886-434e-8c4a-06f73f033576" containerName="console"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489774    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1ff1aaa-b886-434e-8c4a-06f73f033576" containerName="console"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489783    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="config-reloader"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489789    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="config-reloader"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489795    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="kube-rbac-proxy"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489800    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="kube-rbac-proxy"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489846    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="845be81c-3808-4bd3-90bb-fc9b3a4a198c" containerName="console"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489855    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b883f9-df2b-4c15-ab55-79a6b927cd97" containerName="console"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489861    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="48cf9622-0adf-4129-955f-3e82431ab4ab" containerName="registry"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489867    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="kube-rbac-proxy-metric"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489873    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="alertmanager"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489878    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="kube-rbac-proxy-web"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489884    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1ff1aaa-b886-434e-8c4a-06f73f033576" containerName="console"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489888    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="prom-label-proxy"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489894    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="kube-rbac-proxy"
Sep 09 05:52:26.489915 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.489899    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" containerName="config-reloader"
Sep 09 05:52:26.494880 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.494864    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.497184 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.497155    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 05:52:26.497281 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.497181    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 05:52:26.497437 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.497421    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 05:52:26.497538 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.497507    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 05:52:26.497607 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.497542    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 05:52:26.497607 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.497554    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 05:52:26.497607 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.497596    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cftx8\""
Sep 09 05:52:26.497837 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.497822    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 05:52:26.497896 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.497828    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 05:52:26.506274 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.504087    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 05:52:26.510525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.510504    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 05:52:26.532245 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.532219    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15feeca5-4318-462d-a202-84fde3be6099-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.532351 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.532253    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15feeca5-4318-462d-a202-84fde3be6099-tls-assets\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.532351 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.532271    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.532351 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.532293    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.532351 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.532334    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-config-volume\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.532542 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.532352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gwxq\" (UniqueName: \"kubernetes.io/projected/15feeca5-4318-462d-a202-84fde3be6099-kube-api-access-2gwxq\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.532542 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.532421    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15feeca5-4318-462d-a202-84fde3be6099-config-out\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.532542 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.532468    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15feeca5-4318-462d-a202-84fde3be6099-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.532640 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.532553    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.532640 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.532583    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.532640 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.532615    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15feeca5-4318-462d-a202-84fde3be6099-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.532764 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.532663    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-web-config\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.532764 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.532690    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.633177 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.633145    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.633177 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.633177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-config-volume\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.633412 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.633194    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gwxq\" (UniqueName: \"kubernetes.io/projected/15feeca5-4318-462d-a202-84fde3be6099-kube-api-access-2gwxq\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.633412 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.633212    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15feeca5-4318-462d-a202-84fde3be6099-config-out\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.633412 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.633233    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15feeca5-4318-462d-a202-84fde3be6099-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.633412 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.633270    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.633629 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.633409    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.633629 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.633478    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15feeca5-4318-462d-a202-84fde3be6099-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.633629 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.633571    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-web-config\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.633629 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.633614    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.633840 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.633652    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15feeca5-4318-462d-a202-84fde3be6099-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.633840 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.633685    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15feeca5-4318-462d-a202-84fde3be6099-tls-assets\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.633840 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.633710    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.634095 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.634061    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15feeca5-4318-462d-a202-84fde3be6099-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.634582 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.634557    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15feeca5-4318-462d-a202-84fde3be6099-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.634741 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.634719    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15feeca5-4318-462d-a202-84fde3be6099-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.636298 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.636271    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15feeca5-4318-462d-a202-84fde3be6099-config-out\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.636734 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.636711    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.636832 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.636810    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.637032 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.637011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.637066 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.637029    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-web-config\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.637066 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.637018    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.637128 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.637065    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15feeca5-4318-462d-a202-84fde3be6099-tls-assets\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.637687 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.637672    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.637777 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.637760    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15feeca5-4318-462d-a202-84fde3be6099-config-volume\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.641602 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.641585    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gwxq\" (UniqueName: \"kubernetes.io/projected/15feeca5-4318-462d-a202-84fde3be6099-kube-api-access-2gwxq\") pod \"alertmanager-main-0\" (UID: \"15feeca5-4318-462d-a202-84fde3be6099\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:26.807108 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:26.807014    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 05:52:27.086052 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:27.086024    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 05:52:27.087401 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:52:27.087375    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15feeca5_4318_462d_a202_84fde3be6099.slice/crio-4f2215c56db03a69fe5a660d3a9665b22ab5f4ee6985500edfdfe3a3f548e606 WatchSource:0}: Error finding container 4f2215c56db03a69fe5a660d3a9665b22ab5f4ee6985500edfdfe3a3f548e606: Status 404 returned error can't find the container with id 4f2215c56db03a69fe5a660d3a9665b22ab5f4ee6985500edfdfe3a3f548e606
Sep 09 05:52:27.142773 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:27.142744    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15feeca5-4318-462d-a202-84fde3be6099","Type":"ContainerStarted","Data":"4f2215c56db03a69fe5a660d3a9665b22ab5f4ee6985500edfdfe3a3f548e606"}
Sep 09 05:52:27.301054 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:27.301019    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d98f6c8-6c49-44df-8610-2583c98be150" path="/var/lib/kubelet/pods/5d98f6c8-6c49-44df-8610-2583c98be150/volumes"
Sep 09 05:52:28.148749 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.148717    2564 generic.go:356] "Generic (PLEG): container finished" podID="15feeca5-4318-462d-a202-84fde3be6099" containerID="0df0021a87ac60b141c562962bd8093edb2fe7fce0c38e82694ef27634cc9b52" exitCode=0
Sep 09 05:52:28.149116 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.148781    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15feeca5-4318-462d-a202-84fde3be6099","Type":"ContainerDied","Data":"0df0021a87ac60b141c562962bd8093edb2fe7fce0c38e82694ef27634cc9b52"}
Sep 09 05:52:28.561825 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.561793    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59c7bb8db9-hdr9s"]
Sep 09 05:52:28.565082 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.565059    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.577930 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.577901    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59c7bb8db9-hdr9s"]
Sep 09 05:52:28.651317 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.651280    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-console-config\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.651486 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.651327    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-oauth-serving-cert\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.651486 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.651404    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57aed479-fc13-4425-99e0-83815dd3172c-console-oauth-config\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.651486 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.651445    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57aed479-fc13-4425-99e0-83815dd3172c-console-serving-cert\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.651702 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.651590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-service-ca\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.651702 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.651619    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtkfm\" (UniqueName: \"kubernetes.io/projected/57aed479-fc13-4425-99e0-83815dd3172c-kube-api-access-dtkfm\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.651702 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.651645    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-trusted-ca-bundle\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.752851 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.752763    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57aed479-fc13-4425-99e0-83815dd3172c-console-oauth-config\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.752851 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.752806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57aed479-fc13-4425-99e0-83815dd3172c-console-serving-cert\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.752851 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.752852    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-service-ca\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.753106 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.752867    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtkfm\" (UniqueName: \"kubernetes.io/projected/57aed479-fc13-4425-99e0-83815dd3172c-kube-api-access-dtkfm\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.753106 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.752886    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-trusted-ca-bundle\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.753106 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.752905    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-console-config\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.753106 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.752927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-oauth-serving-cert\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.753746 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.753721    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-service-ca\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.753853 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.753765    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-console-config\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.753853 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.753765    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-oauth-serving-cert\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.753990 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.753967    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-trusted-ca-bundle\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.755285 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.755258    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57aed479-fc13-4425-99e0-83815dd3172c-console-serving-cert\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.755372 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.755346    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57aed479-fc13-4425-99e0-83815dd3172c-console-oauth-config\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.760299 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.760281    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtkfm\" (UniqueName: \"kubernetes.io/projected/57aed479-fc13-4425-99e0-83815dd3172c-kube-api-access-dtkfm\") pod \"console-59c7bb8db9-hdr9s\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") " pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:28.875398 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:28.875360    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:29.146397 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:29.146372    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59c7bb8db9-hdr9s"]
Sep 09 05:52:29.148912 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:52:29.148885    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57aed479_fc13_4425_99e0_83815dd3172c.slice/crio-1b3eaf21ffe7d15ecdcfb066260f1fbefd5eca16ee6eaf52952266bcd4efa87c WatchSource:0}: Error finding container 1b3eaf21ffe7d15ecdcfb066260f1fbefd5eca16ee6eaf52952266bcd4efa87c: Status 404 returned error can't find the container with id 1b3eaf21ffe7d15ecdcfb066260f1fbefd5eca16ee6eaf52952266bcd4efa87c
Sep 09 05:52:29.162960 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:29.162928    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15feeca5-4318-462d-a202-84fde3be6099","Type":"ContainerStarted","Data":"4992790fc4231be98efaec8d03190c00ca21e0919656ccf2cceab535ddf909ec"}
Sep 09 05:52:29.163052 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:29.163036    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15feeca5-4318-462d-a202-84fde3be6099","Type":"ContainerStarted","Data":"52cb2565e79447404664ad4c487d74beb21f41c435f47811757ac6a7e8e569b4"}
Sep 09 05:52:29.163091 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:29.163082    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15feeca5-4318-462d-a202-84fde3be6099","Type":"ContainerStarted","Data":"9c427c3ac22523fa3adfaa3ac72453c1c710e92584d936224e8befef7147a9d1"}
Sep 09 05:52:29.163125 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:29.163118    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15feeca5-4318-462d-a202-84fde3be6099","Type":"ContainerStarted","Data":"b3af0e34618617b7bda94ab89f96ead109b3e5400c5db6a8ac548a3c0cc6273e"}
Sep 09 05:52:29.163162 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:29.163153    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15feeca5-4318-462d-a202-84fde3be6099","Type":"ContainerStarted","Data":"032ed07a07d0998f2db15f04c8cac696e105cf7f00a50aec65283755bba71bd8"}
Sep 09 05:52:29.163202 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:29.163190    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15feeca5-4318-462d-a202-84fde3be6099","Type":"ContainerStarted","Data":"4f94846207b3e82f272fccb5f680e69b9ed78bca41c4bf351e12bd47dfcd8168"}
Sep 09 05:52:29.189212 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:29.189155    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.168520984 podStartE2EDuration="3.189138751s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.617767ms" isStatefulPod=false podCreationTimestamp="2026-09-09 05:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:52:29.187355624 +0000 UTC m=+176.403685544" watchObservedRunningTime="2026-09-09 05:52:29.189138751 +0000 UTC m=+176.405468634"
Sep 09 05:52:30.169191 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:30.169152    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c7bb8db9-hdr9s" event={"ID":"57aed479-fc13-4425-99e0-83815dd3172c","Type":"ContainerStarted","Data":"14c5a5062287d9c310fd657ad9c69fed9365bdb05d7c2970a9f67ce541787bd7"}
Sep 09 05:52:30.169191 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:30.169201    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c7bb8db9-hdr9s" event={"ID":"57aed479-fc13-4425-99e0-83815dd3172c","Type":"ContainerStarted","Data":"1b3eaf21ffe7d15ecdcfb066260f1fbefd5eca16ee6eaf52952266bcd4efa87c"}
Sep 09 05:52:38.876521 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:38.876460    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:38.876912 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:38.876576    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:38.881123 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:38.881100    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:38.898858 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:38.898808    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59c7bb8db9-hdr9s" podStartSLOduration=10.898794977 podStartE2EDuration="10.898794977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:52:30.185243515 +0000 UTC m=+177.401573401" watchObservedRunningTime="2026-09-09 05:52:38.898794977 +0000 UTC m=+186.115124860"
Sep 09 05:52:39.209352 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:39.209268    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:52:39.251120 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:52:39.251084    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd8996fdc-chl5s"]
Sep 09 05:53:04.275093 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.275029    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cd8996fdc-chl5s" podUID="33277503-645e-402a-9629-c689c9a2df1f" containerName="console" containerID="cri-o://483213fbdfe84466dd88fc960e29e96aef85efddf8f7414abd6e9582534ef8cf" gracePeriod=15
Sep 09 05:53:04.525613 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.525542    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:53:04.649317 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.649277    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-oauth-serving-cert\") pod \"33277503-645e-402a-9629-c689c9a2df1f\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") "
Sep 09 05:53:04.649317 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.649322    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-console-config\") pod \"33277503-645e-402a-9629-c689c9a2df1f\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") "
Sep 09 05:53:04.649587 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.649412    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-service-ca\") pod \"33277503-645e-402a-9629-c689c9a2df1f\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") "
Sep 09 05:53:04.649587 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.649438    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg86p\" (UniqueName: \"kubernetes.io/projected/33277503-645e-402a-9629-c689c9a2df1f-kube-api-access-fg86p\") pod \"33277503-645e-402a-9629-c689c9a2df1f\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") "
Sep 09 05:53:04.649587 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.649461    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-trusted-ca-bundle\") pod \"33277503-645e-402a-9629-c689c9a2df1f\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") "
Sep 09 05:53:04.649587 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.649520    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33277503-645e-402a-9629-c689c9a2df1f-console-serving-cert\") pod \"33277503-645e-402a-9629-c689c9a2df1f\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") "
Sep 09 05:53:04.649587 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.649568    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33277503-645e-402a-9629-c689c9a2df1f-console-oauth-config\") pod \"33277503-645e-402a-9629-c689c9a2df1f\" (UID: \"33277503-645e-402a-9629-c689c9a2df1f\") "
Sep 09 05:53:04.649830 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.649792    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "33277503-645e-402a-9629-c689c9a2df1f" (UID: "33277503-645e-402a-9629-c689c9a2df1f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:53:04.649922 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.649891    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-console-config" (OuterVolumeSpecName: "console-config") pod "33277503-645e-402a-9629-c689c9a2df1f" (UID: "33277503-645e-402a-9629-c689c9a2df1f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:53:04.649986 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.649920    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-service-ca" (OuterVolumeSpecName: "service-ca") pod "33277503-645e-402a-9629-c689c9a2df1f" (UID: "33277503-645e-402a-9629-c689c9a2df1f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:53:04.649986 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.649946    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "33277503-645e-402a-9629-c689c9a2df1f" (UID: "33277503-645e-402a-9629-c689c9a2df1f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:53:04.651700 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.651671    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33277503-645e-402a-9629-c689c9a2df1f-kube-api-access-fg86p" (OuterVolumeSpecName: "kube-api-access-fg86p") pod "33277503-645e-402a-9629-c689c9a2df1f" (UID: "33277503-645e-402a-9629-c689c9a2df1f"). InnerVolumeSpecName "kube-api-access-fg86p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:53:04.651700 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.651693    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33277503-645e-402a-9629-c689c9a2df1f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "33277503-645e-402a-9629-c689c9a2df1f" (UID: "33277503-645e-402a-9629-c689c9a2df1f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:53:04.651839 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.651670    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33277503-645e-402a-9629-c689c9a2df1f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "33277503-645e-402a-9629-c689c9a2df1f" (UID: "33277503-645e-402a-9629-c689c9a2df1f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:53:04.750406 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.750371    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33277503-645e-402a-9629-c689c9a2df1f-console-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:53:04.750406 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.750399    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33277503-645e-402a-9629-c689c9a2df1f-console-oauth-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:53:04.750406 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.750411    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-oauth-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:53:04.750643 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.750420    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-console-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:53:04.750643 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.750429    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:53:04.750643 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.750438    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg86p\" (UniqueName: \"kubernetes.io/projected/33277503-645e-402a-9629-c689c9a2df1f-kube-api-access-fg86p\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:53:04.750643 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:04.750446    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33277503-645e-402a-9629-c689c9a2df1f-trusted-ca-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:53:05.307337 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:05.307302    2564 generic.go:356] "Generic (PLEG): container finished" podID="33277503-645e-402a-9629-c689c9a2df1f" containerID="483213fbdfe84466dd88fc960e29e96aef85efddf8f7414abd6e9582534ef8cf" exitCode=0
Sep 09 05:53:05.307769 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:05.307346    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd8996fdc-chl5s"
Sep 09 05:53:05.307769 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:05.307380    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd8996fdc-chl5s" event={"ID":"33277503-645e-402a-9629-c689c9a2df1f","Type":"ContainerDied","Data":"483213fbdfe84466dd88fc960e29e96aef85efddf8f7414abd6e9582534ef8cf"}
Sep 09 05:53:05.307769 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:05.307423    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd8996fdc-chl5s" event={"ID":"33277503-645e-402a-9629-c689c9a2df1f","Type":"ContainerDied","Data":"f2f544c65fd5ee9bb8d3fffce25ea1ee89f0d831c263bd81c72d397d8cae2b1d"}
Sep 09 05:53:05.307769 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:05.307443    2564 scope.go:117] "RemoveContainer" containerID="483213fbdfe84466dd88fc960e29e96aef85efddf8f7414abd6e9582534ef8cf"
Sep 09 05:53:05.315696 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:05.315678    2564 scope.go:117] "RemoveContainer" containerID="483213fbdfe84466dd88fc960e29e96aef85efddf8f7414abd6e9582534ef8cf"
Sep 09 05:53:05.315964 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:53:05.315944    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"483213fbdfe84466dd88fc960e29e96aef85efddf8f7414abd6e9582534ef8cf\": container with ID starting with 483213fbdfe84466dd88fc960e29e96aef85efddf8f7414abd6e9582534ef8cf not found: ID does not exist" containerID="483213fbdfe84466dd88fc960e29e96aef85efddf8f7414abd6e9582534ef8cf"
Sep 09 05:53:05.316027 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:05.315973    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"483213fbdfe84466dd88fc960e29e96aef85efddf8f7414abd6e9582534ef8cf"} err="failed to get container status \"483213fbdfe84466dd88fc960e29e96aef85efddf8f7414abd6e9582534ef8cf\": rpc error: code = NotFound desc = could not find container \"483213fbdfe84466dd88fc960e29e96aef85efddf8f7414abd6e9582534ef8cf\": container with ID starting with 483213fbdfe84466dd88fc960e29e96aef85efddf8f7414abd6e9582534ef8cf not found: ID does not exist"
Sep 09 05:53:05.325280 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:05.325258    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd8996fdc-chl5s"]
Sep 09 05:53:05.332528 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:05.332508    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cd8996fdc-chl5s"]
Sep 09 05:53:07.300938 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:07.300904    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33277503-645e-402a-9629-c689c9a2df1f" path="/var/lib/kubelet/pods/33277503-645e-402a-9629-c689c9a2df1f/volumes"
Sep 09 05:53:38.036642 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.036600    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"]
Sep 09 05:53:38.037179 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.036897    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33277503-645e-402a-9629-c689c9a2df1f" containerName="console"
Sep 09 05:53:38.037179 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.036909    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="33277503-645e-402a-9629-c689c9a2df1f" containerName="console"
Sep 09 05:53:38.037179 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.036974    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="33277503-645e-402a-9629-c689c9a2df1f" containerName="console"
Sep 09 05:53:38.041370 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.041352    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"
Sep 09 05:53:38.043749 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.043723    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 05:53:38.043875 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.043730    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xrhv9\""
Sep 09 05:53:38.044550 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.044534    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 05:53:38.049066 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.049038    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"]
Sep 09 05:53:38.112551 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.112520    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c419bb5-f645-4eaa-95e7-23b00d70b604-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn\" (UID: \"2c419bb5-f645-4eaa-95e7-23b00d70b604\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"
Sep 09 05:53:38.112711 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.112574    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c419bb5-f645-4eaa-95e7-23b00d70b604-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn\" (UID: \"2c419bb5-f645-4eaa-95e7-23b00d70b604\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"
Sep 09 05:53:38.112711 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.112631    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r22fp\" (UniqueName: \"kubernetes.io/projected/2c419bb5-f645-4eaa-95e7-23b00d70b604-kube-api-access-r22fp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn\" (UID: \"2c419bb5-f645-4eaa-95e7-23b00d70b604\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"
Sep 09 05:53:38.213705 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.213668    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c419bb5-f645-4eaa-95e7-23b00d70b604-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn\" (UID: \"2c419bb5-f645-4eaa-95e7-23b00d70b604\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"
Sep 09 05:53:38.213864 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.213743    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r22fp\" (UniqueName: \"kubernetes.io/projected/2c419bb5-f645-4eaa-95e7-23b00d70b604-kube-api-access-r22fp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn\" (UID: \"2c419bb5-f645-4eaa-95e7-23b00d70b604\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"
Sep 09 05:53:38.213864 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.213764    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c419bb5-f645-4eaa-95e7-23b00d70b604-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn\" (UID: \"2c419bb5-f645-4eaa-95e7-23b00d70b604\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"
Sep 09 05:53:38.214051 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.214032    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c419bb5-f645-4eaa-95e7-23b00d70b604-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn\" (UID: \"2c419bb5-f645-4eaa-95e7-23b00d70b604\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"
Sep 09 05:53:38.214091 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.214061    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c419bb5-f645-4eaa-95e7-23b00d70b604-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn\" (UID: \"2c419bb5-f645-4eaa-95e7-23b00d70b604\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"
Sep 09 05:53:38.222300 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.222274    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r22fp\" (UniqueName: \"kubernetes.io/projected/2c419bb5-f645-4eaa-95e7-23b00d70b604-kube-api-access-r22fp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn\" (UID: \"2c419bb5-f645-4eaa-95e7-23b00d70b604\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"
Sep 09 05:53:38.350419 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.350373    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"
Sep 09 05:53:38.620959 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:38.620884    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"]
Sep 09 05:53:38.624517 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:53:38.624470    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c419bb5_f645_4eaa_95e7_23b00d70b604.slice/crio-5d73e22e927465935fcd1b04a22e7758f9137b7d9432f2fb22f39b1c8e2b2b9c WatchSource:0}: Error finding container 5d73e22e927465935fcd1b04a22e7758f9137b7d9432f2fb22f39b1c8e2b2b9c: Status 404 returned error can't find the container with id 5d73e22e927465935fcd1b04a22e7758f9137b7d9432f2fb22f39b1c8e2b2b9c
Sep 09 05:53:39.444031 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:39.443994    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn" event={"ID":"2c419bb5-f645-4eaa-95e7-23b00d70b604","Type":"ContainerStarted","Data":"5d73e22e927465935fcd1b04a22e7758f9137b7d9432f2fb22f39b1c8e2b2b9c"}
Sep 09 05:53:44.467615 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:44.467580    2564 generic.go:356] "Generic (PLEG): container finished" podID="2c419bb5-f645-4eaa-95e7-23b00d70b604" containerID="ee4a828ec9a6b463ec78198a1ef4f7eb546155c54f2d49f3ee711dbe4b959210" exitCode=0
Sep 09 05:53:44.467972 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:44.467629    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn" event={"ID":"2c419bb5-f645-4eaa-95e7-23b00d70b604","Type":"ContainerDied","Data":"ee4a828ec9a6b463ec78198a1ef4f7eb546155c54f2d49f3ee711dbe4b959210"}
Sep 09 05:53:46.478369 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:46.478330    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn" event={"ID":"2c419bb5-f645-4eaa-95e7-23b00d70b604","Type":"ContainerStarted","Data":"a9b36e47b52a4f6a3fda340a2172b77ff081623b0687e1d9218ace437bf44dc8"}
Sep 09 05:53:47.483588 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:47.483547    2564 generic.go:356] "Generic (PLEG): container finished" podID="2c419bb5-f645-4eaa-95e7-23b00d70b604" containerID="a9b36e47b52a4f6a3fda340a2172b77ff081623b0687e1d9218ace437bf44dc8" exitCode=0
Sep 09 05:53:47.483957 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:47.483628    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn" event={"ID":"2c419bb5-f645-4eaa-95e7-23b00d70b604","Type":"ContainerDied","Data":"a9b36e47b52a4f6a3fda340a2172b77ff081623b0687e1d9218ace437bf44dc8"}
Sep 09 05:53:54.518176 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:54.518139    2564 generic.go:356] "Generic (PLEG): container finished" podID="2c419bb5-f645-4eaa-95e7-23b00d70b604" containerID="dc173dff3321f0bfbe06b3c196498ffb20c689b7f5d9026cb1ad29e73971a900" exitCode=0
Sep 09 05:53:54.518571 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:54.518222    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn" event={"ID":"2c419bb5-f645-4eaa-95e7-23b00d70b604","Type":"ContainerDied","Data":"dc173dff3321f0bfbe06b3c196498ffb20c689b7f5d9026cb1ad29e73971a900"}
Sep 09 05:53:55.645009 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:55.644986    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"
Sep 09 05:53:55.767935 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:55.767895    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c419bb5-f645-4eaa-95e7-23b00d70b604-bundle\") pod \"2c419bb5-f645-4eaa-95e7-23b00d70b604\" (UID: \"2c419bb5-f645-4eaa-95e7-23b00d70b604\") "
Sep 09 05:53:55.768133 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:55.767959    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r22fp\" (UniqueName: \"kubernetes.io/projected/2c419bb5-f645-4eaa-95e7-23b00d70b604-kube-api-access-r22fp\") pod \"2c419bb5-f645-4eaa-95e7-23b00d70b604\" (UID: \"2c419bb5-f645-4eaa-95e7-23b00d70b604\") "
Sep 09 05:53:55.768133 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:55.768051    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c419bb5-f645-4eaa-95e7-23b00d70b604-util\") pod \"2c419bb5-f645-4eaa-95e7-23b00d70b604\" (UID: \"2c419bb5-f645-4eaa-95e7-23b00d70b604\") "
Sep 09 05:53:55.768647 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:55.768622    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c419bb5-f645-4eaa-95e7-23b00d70b604-bundle" (OuterVolumeSpecName: "bundle") pod "2c419bb5-f645-4eaa-95e7-23b00d70b604" (UID: "2c419bb5-f645-4eaa-95e7-23b00d70b604"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:53:55.770210 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:55.770182    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c419bb5-f645-4eaa-95e7-23b00d70b604-kube-api-access-r22fp" (OuterVolumeSpecName: "kube-api-access-r22fp") pod "2c419bb5-f645-4eaa-95e7-23b00d70b604" (UID: "2c419bb5-f645-4eaa-95e7-23b00d70b604"). InnerVolumeSpecName "kube-api-access-r22fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:53:55.772197 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:55.772164    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c419bb5-f645-4eaa-95e7-23b00d70b604-util" (OuterVolumeSpecName: "util") pod "2c419bb5-f645-4eaa-95e7-23b00d70b604" (UID: "2c419bb5-f645-4eaa-95e7-23b00d70b604"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:53:55.868827 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:55.868793    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c419bb5-f645-4eaa-95e7-23b00d70b604-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:53:55.868827 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:55.868821    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r22fp\" (UniqueName: \"kubernetes.io/projected/2c419bb5-f645-4eaa-95e7-23b00d70b604-kube-api-access-r22fp\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:53:55.868827 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:55.868831    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c419bb5-f645-4eaa-95e7-23b00d70b604-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:53:56.528960 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:56.528925    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn" event={"ID":"2c419bb5-f645-4eaa-95e7-23b00d70b604","Type":"ContainerDied","Data":"5d73e22e927465935fcd1b04a22e7758f9137b7d9432f2fb22f39b1c8e2b2b9c"}
Sep 09 05:53:56.528960 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:56.528957    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409fqdn"
Sep 09 05:53:56.528960 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:53:56.528968    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d73e22e927465935fcd1b04a22e7758f9137b7d9432f2fb22f39b1c8e2b2b9c"
Sep 09 05:54:01.179236 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.179202    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-58cnw"]
Sep 09 05:54:01.179700 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.179539    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c419bb5-f645-4eaa-95e7-23b00d70b604" containerName="util"
Sep 09 05:54:01.179700 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.179552    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c419bb5-f645-4eaa-95e7-23b00d70b604" containerName="util"
Sep 09 05:54:01.179700 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.179569    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c419bb5-f645-4eaa-95e7-23b00d70b604" containerName="extract"
Sep 09 05:54:01.179700 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.179575    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c419bb5-f645-4eaa-95e7-23b00d70b604" containerName="extract"
Sep 09 05:54:01.179700 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.179591    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c419bb5-f645-4eaa-95e7-23b00d70b604" containerName="pull"
Sep 09 05:54:01.179700 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.179597    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c419bb5-f645-4eaa-95e7-23b00d70b604" containerName="pull"
Sep 09 05:54:01.179700 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.179650    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c419bb5-f645-4eaa-95e7-23b00d70b604" containerName="extract"
Sep 09 05:54:01.206685 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.206660    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-58cnw"]
Sep 09 05:54:01.206844 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.206752    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-58cnw"
Sep 09 05:54:01.209614 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.209586    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 09 05:54:01.209769 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.209612    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 09 05:54:01.209769 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.209632    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-bmmkm\""
Sep 09 05:54:01.312103 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.312074    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d33de6f-151f-48c3-b8ff-c012ba28a286-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-58cnw\" (UID: \"6d33de6f-151f-48c3-b8ff-c012ba28a286\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-58cnw"
Sep 09 05:54:01.312103 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.312105    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4qz\" (UniqueName: \"kubernetes.io/projected/6d33de6f-151f-48c3-b8ff-c012ba28a286-kube-api-access-dc4qz\") pod \"cert-manager-operator-controller-manager-55c444bf86-58cnw\" (UID: \"6d33de6f-151f-48c3-b8ff-c012ba28a286\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-58cnw"
Sep 09 05:54:01.413022 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.412981    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d33de6f-151f-48c3-b8ff-c012ba28a286-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-58cnw\" (UID: \"6d33de6f-151f-48c3-b8ff-c012ba28a286\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-58cnw"
Sep 09 05:54:01.413022 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.413020    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4qz\" (UniqueName: \"kubernetes.io/projected/6d33de6f-151f-48c3-b8ff-c012ba28a286-kube-api-access-dc4qz\") pod \"cert-manager-operator-controller-manager-55c444bf86-58cnw\" (UID: \"6d33de6f-151f-48c3-b8ff-c012ba28a286\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-58cnw"
Sep 09 05:54:01.413437 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.413414    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d33de6f-151f-48c3-b8ff-c012ba28a286-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-58cnw\" (UID: \"6d33de6f-151f-48c3-b8ff-c012ba28a286\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-58cnw"
Sep 09 05:54:01.421906 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.421875    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4qz\" (UniqueName: \"kubernetes.io/projected/6d33de6f-151f-48c3-b8ff-c012ba28a286-kube-api-access-dc4qz\") pod \"cert-manager-operator-controller-manager-55c444bf86-58cnw\" (UID: \"6d33de6f-151f-48c3-b8ff-c012ba28a286\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-58cnw"
Sep 09 05:54:01.516645 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.516563    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-58cnw"
Sep 09 05:54:01.812482 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:01.812394    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-58cnw"]
Sep 09 05:54:01.816662 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:54:01.816623    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d33de6f_151f_48c3_b8ff_c012ba28a286.slice/crio-134ceef4294d60360283d96681fc39c1a90d30583baa3dffd66e152a0fdf3526 WatchSource:0}: Error finding container 134ceef4294d60360283d96681fc39c1a90d30583baa3dffd66e152a0fdf3526: Status 404 returned error can't find the container with id 134ceef4294d60360283d96681fc39c1a90d30583baa3dffd66e152a0fdf3526
Sep 09 05:54:02.553564 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:02.553524    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-58cnw" event={"ID":"6d33de6f-151f-48c3-b8ff-c012ba28a286","Type":"ContainerStarted","Data":"134ceef4294d60360283d96681fc39c1a90d30583baa3dffd66e152a0fdf3526"}
Sep 09 05:54:04.565889 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:04.565141    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-58cnw" event={"ID":"6d33de6f-151f-48c3-b8ff-c012ba28a286","Type":"ContainerStarted","Data":"d38b460611ea1d238a50689b687b01f42b95c379947eb09c7b1768890d66747e"}
Sep 09 05:54:04.586014 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:04.585955    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-58cnw" podStartSLOduration=1.599757495 podStartE2EDuration="3.585936326s" totalImagesPullingTime="1.986178831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:54:04.584661487 +0000 UTC m=+271.800991372" watchObservedRunningTime="2026-09-09 05:54:04.585936326 +0000 UTC m=+271.802266210"
Sep 09 05:54:04.962045 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:04.962014    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-v4n2s"]
Sep 09 05:54:04.965384 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:04.965369    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-v4n2s"
Sep 09 05:54:04.968217 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:04.968197    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 05:54:04.968976 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:04.968960    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 05:54:04.969078 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:04.968986    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ddnrm\""
Sep 09 05:54:04.984069 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:04.984044    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-v4n2s"]
Sep 09 05:54:05.145535 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.145469    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6cfp\" (UniqueName: \"kubernetes.io/projected/1f08bfd0-259d-4b05-89f7-6e57086822d6-kube-api-access-q6cfp\") pod \"cert-manager-webhook-6486ffd997-v4n2s\" (UID: \"1f08bfd0-259d-4b05-89f7-6e57086822d6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v4n2s"
Sep 09 05:54:05.145706 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.145553    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f08bfd0-259d-4b05-89f7-6e57086822d6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-v4n2s\" (UID: \"1f08bfd0-259d-4b05-89f7-6e57086822d6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v4n2s"
Sep 09 05:54:05.246283 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.246192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f08bfd0-259d-4b05-89f7-6e57086822d6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-v4n2s\" (UID: \"1f08bfd0-259d-4b05-89f7-6e57086822d6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v4n2s"
Sep 09 05:54:05.246283 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.246280    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6cfp\" (UniqueName: \"kubernetes.io/projected/1f08bfd0-259d-4b05-89f7-6e57086822d6-kube-api-access-q6cfp\") pod \"cert-manager-webhook-6486ffd997-v4n2s\" (UID: \"1f08bfd0-259d-4b05-89f7-6e57086822d6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v4n2s"
Sep 09 05:54:05.264432 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.264403    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f08bfd0-259d-4b05-89f7-6e57086822d6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-v4n2s\" (UID: \"1f08bfd0-259d-4b05-89f7-6e57086822d6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v4n2s"
Sep 09 05:54:05.264619 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.264600    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6cfp\" (UniqueName: \"kubernetes.io/projected/1f08bfd0-259d-4b05-89f7-6e57086822d6-kube-api-access-q6cfp\") pod \"cert-manager-webhook-6486ffd997-v4n2s\" (UID: \"1f08bfd0-259d-4b05-89f7-6e57086822d6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v4n2s"
Sep 09 05:54:05.283430 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.283403    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-v4n2s"
Sep 09 05:54:05.553062 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.553025    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-v4n2s"]
Sep 09 05:54:05.555139 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:54:05.555110    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f08bfd0_259d_4b05_89f7_6e57086822d6.slice/crio-408defba3d738382a9ff651684203d37db324b27a0d278ced203ebb58f16b8a0 WatchSource:0}: Error finding container 408defba3d738382a9ff651684203d37db324b27a0d278ced203ebb58f16b8a0: Status 404 returned error can't find the container with id 408defba3d738382a9ff651684203d37db324b27a0d278ced203ebb58f16b8a0
Sep 09 05:54:05.570950 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.570919    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-v4n2s" event={"ID":"1f08bfd0-259d-4b05-89f7-6e57086822d6","Type":"ContainerStarted","Data":"408defba3d738382a9ff651684203d37db324b27a0d278ced203ebb58f16b8a0"}
Sep 09 05:54:05.672235 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.672202    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"]
Sep 09 05:54:05.676762 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.676743    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"
Sep 09 05:54:05.679924 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.679903    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 05:54:05.680027 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.679968    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xrhv9\""
Sep 09 05:54:05.680370 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.680356    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 05:54:05.690141 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.690119    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"]
Sep 09 05:54:05.851043 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.851010    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d1fefcb-2dbe-47d8-9586-4708705c4505-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg\" (UID: \"8d1fefcb-2dbe-47d8-9586-4708705c4505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"
Sep 09 05:54:05.851199 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.851062    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d1fefcb-2dbe-47d8-9586-4708705c4505-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg\" (UID: \"8d1fefcb-2dbe-47d8-9586-4708705c4505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"
Sep 09 05:54:05.851199 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.851081    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bxlw\" (UniqueName: \"kubernetes.io/projected/8d1fefcb-2dbe-47d8-9586-4708705c4505-kube-api-access-6bxlw\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg\" (UID: \"8d1fefcb-2dbe-47d8-9586-4708705c4505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"
Sep 09 05:54:05.952321 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.952285    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d1fefcb-2dbe-47d8-9586-4708705c4505-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg\" (UID: \"8d1fefcb-2dbe-47d8-9586-4708705c4505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"
Sep 09 05:54:05.952529 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.952346    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d1fefcb-2dbe-47d8-9586-4708705c4505-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg\" (UID: \"8d1fefcb-2dbe-47d8-9586-4708705c4505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"
Sep 09 05:54:05.952529 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.952377    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxlw\" (UniqueName: \"kubernetes.io/projected/8d1fefcb-2dbe-47d8-9586-4708705c4505-kube-api-access-6bxlw\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg\" (UID: \"8d1fefcb-2dbe-47d8-9586-4708705c4505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"
Sep 09 05:54:05.952688 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.952670    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d1fefcb-2dbe-47d8-9586-4708705c4505-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg\" (UID: \"8d1fefcb-2dbe-47d8-9586-4708705c4505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"
Sep 09 05:54:05.952726 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.952713    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d1fefcb-2dbe-47d8-9586-4708705c4505-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg\" (UID: \"8d1fefcb-2dbe-47d8-9586-4708705c4505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"
Sep 09 05:54:05.960471 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.960454    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxlw\" (UniqueName: \"kubernetes.io/projected/8d1fefcb-2dbe-47d8-9586-4708705c4505-kube-api-access-6bxlw\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg\" (UID: \"8d1fefcb-2dbe-47d8-9586-4708705c4505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"
Sep 09 05:54:05.985814 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:05.985791    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"
Sep 09 05:54:06.254251 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:06.254220    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"]
Sep 09 05:54:06.256804 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:54:06.256777    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1fefcb_2dbe_47d8_9586_4708705c4505.slice/crio-0cb83c46ef24dfa9f0ceae311d9f1a1ae43c2e4271d7300f9b933c730a663094 WatchSource:0}: Error finding container 0cb83c46ef24dfa9f0ceae311d9f1a1ae43c2e4271d7300f9b933c730a663094: Status 404 returned error can't find the container with id 0cb83c46ef24dfa9f0ceae311d9f1a1ae43c2e4271d7300f9b933c730a663094
Sep 09 05:54:06.577512 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:06.577460    2564 generic.go:356] "Generic (PLEG): container finished" podID="8d1fefcb-2dbe-47d8-9586-4708705c4505" containerID="38f3cbef6d93a13199db032fe9de475911f73d1049e87de3215b8410dcb042d3" exitCode=0
Sep 09 05:54:06.577987 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:06.577568    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg" event={"ID":"8d1fefcb-2dbe-47d8-9586-4708705c4505","Type":"ContainerDied","Data":"38f3cbef6d93a13199db032fe9de475911f73d1049e87de3215b8410dcb042d3"}
Sep 09 05:54:06.577987 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:06.577614    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg" event={"ID":"8d1fefcb-2dbe-47d8-9586-4708705c4505","Type":"ContainerStarted","Data":"0cb83c46ef24dfa9f0ceae311d9f1a1ae43c2e4271d7300f9b933c730a663094"}
Sep 09 05:54:08.349362 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:08.349326    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ft4b8"]
Sep 09 05:54:08.352942 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:08.352915    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ft4b8"
Sep 09 05:54:08.355318 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:08.355296    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8lwkj\""
Sep 09 05:54:08.361588 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:08.361566    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ft4b8"]
Sep 09 05:54:08.370023 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:08.369964    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx8v8\" (UniqueName: \"kubernetes.io/projected/006ad445-90ac-45e0-b6fe-7a54ef2c7c90-kube-api-access-cx8v8\") pod \"cert-manager-cainjector-559cdd8c87-ft4b8\" (UID: \"006ad445-90ac-45e0-b6fe-7a54ef2c7c90\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ft4b8"
Sep 09 05:54:08.370150 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:08.370024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/006ad445-90ac-45e0-b6fe-7a54ef2c7c90-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ft4b8\" (UID: \"006ad445-90ac-45e0-b6fe-7a54ef2c7c90\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ft4b8"
Sep 09 05:54:08.470715 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:08.470677    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/006ad445-90ac-45e0-b6fe-7a54ef2c7c90-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ft4b8\" (UID: \"006ad445-90ac-45e0-b6fe-7a54ef2c7c90\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ft4b8"
Sep 09 05:54:08.470905 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:08.470818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx8v8\" (UniqueName: \"kubernetes.io/projected/006ad445-90ac-45e0-b6fe-7a54ef2c7c90-kube-api-access-cx8v8\") pod \"cert-manager-cainjector-559cdd8c87-ft4b8\" (UID: \"006ad445-90ac-45e0-b6fe-7a54ef2c7c90\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ft4b8"
Sep 09 05:54:08.478706 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:08.478672    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/006ad445-90ac-45e0-b6fe-7a54ef2c7c90-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ft4b8\" (UID: \"006ad445-90ac-45e0-b6fe-7a54ef2c7c90\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ft4b8"
Sep 09 05:54:08.478917 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:08.478790    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx8v8\" (UniqueName: \"kubernetes.io/projected/006ad445-90ac-45e0-b6fe-7a54ef2c7c90-kube-api-access-cx8v8\") pod \"cert-manager-cainjector-559cdd8c87-ft4b8\" (UID: \"006ad445-90ac-45e0-b6fe-7a54ef2c7c90\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ft4b8"
Sep 09 05:54:08.589719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:08.589633    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-v4n2s" event={"ID":"1f08bfd0-259d-4b05-89f7-6e57086822d6","Type":"ContainerStarted","Data":"32f553afd671f8a627e1ba66646920dcb6539bee28f5be104c86a8bde35c428d"}
Sep 09 05:54:08.589719 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:08.589689    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-v4n2s"
Sep 09 05:54:08.611122 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:08.611021    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-v4n2s" podStartSLOduration=2.128542934 podStartE2EDuration="4.611001881s" totalImagesPullingTime="2.482458947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:54:08.610177174 +0000 UTC m=+275.826507058" watchObservedRunningTime="2026-09-09 05:54:08.611001881 +0000 UTC m=+275.827331770"
Sep 09 05:54:08.664822 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:08.664788    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ft4b8"
Sep 09 05:54:08.945042 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:08.944952    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ft4b8"]
Sep 09 05:54:09.450034 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:54:09.449993    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod006ad445_90ac_45e0_b6fe_7a54ef2c7c90.slice/crio-f8f4f7c83c01591ed3c70abd0585f592cd32be472d5235bd8682cdb126095f27 WatchSource:0}: Error finding container f8f4f7c83c01591ed3c70abd0585f592cd32be472d5235bd8682cdb126095f27: Status 404 returned error can't find the container with id f8f4f7c83c01591ed3c70abd0585f592cd32be472d5235bd8682cdb126095f27
Sep 09 05:54:09.597219 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:09.597182    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg" event={"ID":"8d1fefcb-2dbe-47d8-9586-4708705c4505","Type":"ContainerStarted","Data":"59429bbbf2998b304088186848b65ea77211fe6b742f5ec1f5231e36132cdeb4"}
Sep 09 05:54:09.599599 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:09.599560    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ft4b8" event={"ID":"006ad445-90ac-45e0-b6fe-7a54ef2c7c90","Type":"ContainerStarted","Data":"7e2e8999e090a880ebb2f5ee47dc5ee92668581ff559fbf60bbec150e1bccb70"}
Sep 09 05:54:09.599732 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:09.599600    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ft4b8" event={"ID":"006ad445-90ac-45e0-b6fe-7a54ef2c7c90","Type":"ContainerStarted","Data":"f8f4f7c83c01591ed3c70abd0585f592cd32be472d5235bd8682cdb126095f27"}
Sep 09 05:54:09.630049 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:09.629988    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ft4b8" podStartSLOduration=1.62997027 podStartE2EDuration="1.62997027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:54:09.629379176 +0000 UTC m=+276.845709062" watchObservedRunningTime="2026-09-09 05:54:09.62997027 +0000 UTC m=+276.846300155"
Sep 09 05:54:10.605539 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:10.605503    2564 generic.go:356] "Generic (PLEG): container finished" podID="8d1fefcb-2dbe-47d8-9586-4708705c4505" containerID="59429bbbf2998b304088186848b65ea77211fe6b742f5ec1f5231e36132cdeb4" exitCode=0
Sep 09 05:54:10.605902 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:10.605589    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg" event={"ID":"8d1fefcb-2dbe-47d8-9586-4708705c4505","Type":"ContainerDied","Data":"59429bbbf2998b304088186848b65ea77211fe6b742f5ec1f5231e36132cdeb4"}
Sep 09 05:54:11.611440 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:11.611406    2564 generic.go:356] "Generic (PLEG): container finished" podID="8d1fefcb-2dbe-47d8-9586-4708705c4505" containerID="4c7c7c6ea76a00b76830b5b1851b8057e108d46620894fd3448835a4ae43655b" exitCode=0
Sep 09 05:54:11.611818 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:11.611516    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg" event={"ID":"8d1fefcb-2dbe-47d8-9586-4708705c4505","Type":"ContainerDied","Data":"4c7c7c6ea76a00b76830b5b1851b8057e108d46620894fd3448835a4ae43655b"}
Sep 09 05:54:12.746183 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:12.746161    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"
Sep 09 05:54:12.804619 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:12.804587    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d1fefcb-2dbe-47d8-9586-4708705c4505-bundle\") pod \"8d1fefcb-2dbe-47d8-9586-4708705c4505\" (UID: \"8d1fefcb-2dbe-47d8-9586-4708705c4505\") "
Sep 09 05:54:12.804619 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:12.804624    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bxlw\" (UniqueName: \"kubernetes.io/projected/8d1fefcb-2dbe-47d8-9586-4708705c4505-kube-api-access-6bxlw\") pod \"8d1fefcb-2dbe-47d8-9586-4708705c4505\" (UID: \"8d1fefcb-2dbe-47d8-9586-4708705c4505\") "
Sep 09 05:54:12.804816 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:12.804653    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d1fefcb-2dbe-47d8-9586-4708705c4505-util\") pod \"8d1fefcb-2dbe-47d8-9586-4708705c4505\" (UID: \"8d1fefcb-2dbe-47d8-9586-4708705c4505\") "
Sep 09 05:54:12.805015 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:12.804986    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1fefcb-2dbe-47d8-9586-4708705c4505-bundle" (OuterVolumeSpecName: "bundle") pod "8d1fefcb-2dbe-47d8-9586-4708705c4505" (UID: "8d1fefcb-2dbe-47d8-9586-4708705c4505"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:54:12.806672 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:12.806644    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1fefcb-2dbe-47d8-9586-4708705c4505-kube-api-access-6bxlw" (OuterVolumeSpecName: "kube-api-access-6bxlw") pod "8d1fefcb-2dbe-47d8-9586-4708705c4505" (UID: "8d1fefcb-2dbe-47d8-9586-4708705c4505"). InnerVolumeSpecName "kube-api-access-6bxlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:54:12.809003 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:12.808968    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1fefcb-2dbe-47d8-9586-4708705c4505-util" (OuterVolumeSpecName: "util") pod "8d1fefcb-2dbe-47d8-9586-4708705c4505" (UID: "8d1fefcb-2dbe-47d8-9586-4708705c4505"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:54:12.905239 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:12.905168    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d1fefcb-2dbe-47d8-9586-4708705c4505-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:54:12.905239 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:12.905192    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d1fefcb-2dbe-47d8-9586-4708705c4505-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:54:12.905239 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:12.905202    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bxlw\" (UniqueName: \"kubernetes.io/projected/8d1fefcb-2dbe-47d8-9586-4708705c4505-kube-api-access-6bxlw\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:54:13.622133 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:13.622089    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg" event={"ID":"8d1fefcb-2dbe-47d8-9586-4708705c4505","Type":"ContainerDied","Data":"0cb83c46ef24dfa9f0ceae311d9f1a1ae43c2e4271d7300f9b933c730a663094"}
Sep 09 05:54:13.622133 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:13.622127    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4bqdg"
Sep 09 05:54:13.622324 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:13.622146    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cb83c46ef24dfa9f0ceae311d9f1a1ae43c2e4271d7300f9b933c730a663094"
Sep 09 05:54:14.602000 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:14.601967    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-v4n2s"
Sep 09 05:54:18.864450 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:18.864415    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vk5zx"]
Sep 09 05:54:18.864822 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:18.864726    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1fefcb-2dbe-47d8-9586-4708705c4505" containerName="extract"
Sep 09 05:54:18.864822 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:18.864737    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1fefcb-2dbe-47d8-9586-4708705c4505" containerName="extract"
Sep 09 05:54:18.864822 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:18.864745    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1fefcb-2dbe-47d8-9586-4708705c4505" containerName="util"
Sep 09 05:54:18.864822 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:18.864750    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1fefcb-2dbe-47d8-9586-4708705c4505" containerName="util"
Sep 09 05:54:18.864822 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:18.864764    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1fefcb-2dbe-47d8-9586-4708705c4505" containerName="pull"
Sep 09 05:54:18.864822 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:18.864769    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1fefcb-2dbe-47d8-9586-4708705c4505" containerName="pull"
Sep 09 05:54:18.865000 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:18.864827    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d1fefcb-2dbe-47d8-9586-4708705c4505" containerName="extract"
Sep 09 05:54:18.868500 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:18.868475    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vk5zx"
Sep 09 05:54:18.870977 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:18.870953    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 05:54:18.871795 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:18.871771    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 05:54:18.871795 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:18.871783    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6x6j2\""
Sep 09 05:54:18.874834 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:18.874813    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vk5zx"]
Sep 09 05:54:18.953711 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:18.953677    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsgrk\" (UniqueName: \"kubernetes.io/projected/0b37f545-4ed7-44e5-be4d-9de2f35425bd-kube-api-access-nsgrk\") pod \"openshift-lws-operator-fd8ccff4c-vk5zx\" (UID: \"0b37f545-4ed7-44e5-be4d-9de2f35425bd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vk5zx"
Sep 09 05:54:18.953854 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:18.953736    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b37f545-4ed7-44e5-be4d-9de2f35425bd-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vk5zx\" (UID: \"0b37f545-4ed7-44e5-be4d-9de2f35425bd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vk5zx"
Sep 09 05:54:19.054874 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:19.054842    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b37f545-4ed7-44e5-be4d-9de2f35425bd-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vk5zx\" (UID: \"0b37f545-4ed7-44e5-be4d-9de2f35425bd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vk5zx"
Sep 09 05:54:19.055019 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:19.054892    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsgrk\" (UniqueName: \"kubernetes.io/projected/0b37f545-4ed7-44e5-be4d-9de2f35425bd-kube-api-access-nsgrk\") pod \"openshift-lws-operator-fd8ccff4c-vk5zx\" (UID: \"0b37f545-4ed7-44e5-be4d-9de2f35425bd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vk5zx"
Sep 09 05:54:19.055200 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:19.055179    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b37f545-4ed7-44e5-be4d-9de2f35425bd-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vk5zx\" (UID: \"0b37f545-4ed7-44e5-be4d-9de2f35425bd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vk5zx"
Sep 09 05:54:19.062685 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:19.062660    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsgrk\" (UniqueName: \"kubernetes.io/projected/0b37f545-4ed7-44e5-be4d-9de2f35425bd-kube-api-access-nsgrk\") pod \"openshift-lws-operator-fd8ccff4c-vk5zx\" (UID: \"0b37f545-4ed7-44e5-be4d-9de2f35425bd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vk5zx"
Sep 09 05:54:19.178020 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:19.177944    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vk5zx"
Sep 09 05:54:19.451820 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:19.451795    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vk5zx"]
Sep 09 05:54:19.453792 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:54:19.453763    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b37f545_4ed7_44e5_be4d_9de2f35425bd.slice/crio-6a4e68c65aab5c7b0e4c0fc8834068196aee1afb8087a2f5e90ee65bcd8f09df WatchSource:0}: Error finding container 6a4e68c65aab5c7b0e4c0fc8834068196aee1afb8087a2f5e90ee65bcd8f09df: Status 404 returned error can't find the container with id 6a4e68c65aab5c7b0e4c0fc8834068196aee1afb8087a2f5e90ee65bcd8f09df
Sep 09 05:54:19.649697 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:19.649661    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vk5zx" event={"ID":"0b37f545-4ed7-44e5-be4d-9de2f35425bd","Type":"ContainerStarted","Data":"6a4e68c65aab5c7b0e4c0fc8834068196aee1afb8087a2f5e90ee65bcd8f09df"}
Sep 09 05:54:25.678824 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:25.678782    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vk5zx" event={"ID":"0b37f545-4ed7-44e5-be4d-9de2f35425bd","Type":"ContainerStarted","Data":"d82f07c4456414e079e9df98ffb6f5a87dfae5393240f8e1dc3e4f6381f1b7cb"}
Sep 09 05:54:25.695117 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:25.695068    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vk5zx" podStartSLOduration=2.1783376580000002 podStartE2EDuration="7.695052604s" totalImagesPullingTime="5.516714946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:54:25.693334708 +0000 UTC m=+292.909664593" watchObservedRunningTime="2026-09-09 05:54:25.695052604 +0000 UTC m=+292.911382549"
Sep 09 05:54:27.856331 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:27.856296    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"]
Sep 09 05:54:27.859326 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:27.859304    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"
Sep 09 05:54:27.861592 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:27.861573    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 05:54:27.862583 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:27.862569    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xrhv9\""
Sep 09 05:54:27.862644 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:27.862587    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 05:54:27.866566 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:27.866539    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"]
Sep 09 05:54:27.929345 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:27.929304    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8416ba37-dd88-46c2-bbb1-cb649aeab244-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4\" (UID: \"8416ba37-dd88-46c2-bbb1-cb649aeab244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"
Sep 09 05:54:27.929551 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:27.929371    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7896t\" (UniqueName: \"kubernetes.io/projected/8416ba37-dd88-46c2-bbb1-cb649aeab244-kube-api-access-7896t\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4\" (UID: \"8416ba37-dd88-46c2-bbb1-cb649aeab244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"
Sep 09 05:54:27.929551 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:27.929433    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8416ba37-dd88-46c2-bbb1-cb649aeab244-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4\" (UID: \"8416ba37-dd88-46c2-bbb1-cb649aeab244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"
Sep 09 05:54:28.030317 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:28.030278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7896t\" (UniqueName: \"kubernetes.io/projected/8416ba37-dd88-46c2-bbb1-cb649aeab244-kube-api-access-7896t\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4\" (UID: \"8416ba37-dd88-46c2-bbb1-cb649aeab244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"
Sep 09 05:54:28.030466 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:28.030340    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8416ba37-dd88-46c2-bbb1-cb649aeab244-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4\" (UID: \"8416ba37-dd88-46c2-bbb1-cb649aeab244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"
Sep 09 05:54:28.030466 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:28.030382    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8416ba37-dd88-46c2-bbb1-cb649aeab244-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4\" (UID: \"8416ba37-dd88-46c2-bbb1-cb649aeab244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"
Sep 09 05:54:28.030758 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:28.030743    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8416ba37-dd88-46c2-bbb1-cb649aeab244-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4\" (UID: \"8416ba37-dd88-46c2-bbb1-cb649aeab244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"
Sep 09 05:54:28.030797 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:28.030775    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8416ba37-dd88-46c2-bbb1-cb649aeab244-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4\" (UID: \"8416ba37-dd88-46c2-bbb1-cb649aeab244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"
Sep 09 05:54:28.039221 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:28.039187    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7896t\" (UniqueName: \"kubernetes.io/projected/8416ba37-dd88-46c2-bbb1-cb649aeab244-kube-api-access-7896t\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4\" (UID: \"8416ba37-dd88-46c2-bbb1-cb649aeab244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"
Sep 09 05:54:28.169662 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:28.169581    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"
Sep 09 05:54:28.438109 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:28.438085    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"]
Sep 09 05:54:28.440082 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:54:28.440058    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8416ba37_dd88_46c2_bbb1_cb649aeab244.slice/crio-e212e55c2b303e12fc03bb1415a4e2ce98afa4fc42387383a25c0cacd9b91430 WatchSource:0}: Error finding container e212e55c2b303e12fc03bb1415a4e2ce98afa4fc42387383a25c0cacd9b91430: Status 404 returned error can't find the container with id e212e55c2b303e12fc03bb1415a4e2ce98afa4fc42387383a25c0cacd9b91430
Sep 09 05:54:28.694334 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:28.694249    2564 generic.go:356] "Generic (PLEG): container finished" podID="8416ba37-dd88-46c2-bbb1-cb649aeab244" containerID="078d74f5e8a6cbc87de07f1e37d4acc414e5091ea3c713f827126e61b093a80f" exitCode=0
Sep 09 05:54:28.694475 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:28.694335    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4" event={"ID":"8416ba37-dd88-46c2-bbb1-cb649aeab244","Type":"ContainerDied","Data":"078d74f5e8a6cbc87de07f1e37d4acc414e5091ea3c713f827126e61b093a80f"}
Sep 09 05:54:28.694475 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:28.694385    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4" event={"ID":"8416ba37-dd88-46c2-bbb1-cb649aeab244","Type":"ContainerStarted","Data":"e212e55c2b303e12fc03bb1415a4e2ce98afa4fc42387383a25c0cacd9b91430"}
Sep 09 05:54:29.701651 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:29.701561    2564 generic.go:356] "Generic (PLEG): container finished" podID="8416ba37-dd88-46c2-bbb1-cb649aeab244" containerID="b88fb19c9140d08538caad1f18dc7266f564efe44af3f9386bc27ea29f74a9a6" exitCode=0
Sep 09 05:54:29.701984 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:29.701657    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4" event={"ID":"8416ba37-dd88-46c2-bbb1-cb649aeab244","Type":"ContainerDied","Data":"b88fb19c9140d08538caad1f18dc7266f564efe44af3f9386bc27ea29f74a9a6"}
Sep 09 05:54:30.707891 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:30.707858    2564 generic.go:356] "Generic (PLEG): container finished" podID="8416ba37-dd88-46c2-bbb1-cb649aeab244" containerID="677c70ca0b6e590816ca89004d24c3a427b02e6edef025cabd855a57ba090ecf" exitCode=0
Sep 09 05:54:30.708277 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:30.707945    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4" event={"ID":"8416ba37-dd88-46c2-bbb1-cb649aeab244","Type":"ContainerDied","Data":"677c70ca0b6e590816ca89004d24c3a427b02e6edef025cabd855a57ba090ecf"}
Sep 09 05:54:31.847603 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:31.847579    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"
Sep 09 05:54:31.965241 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:31.965211    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7896t\" (UniqueName: \"kubernetes.io/projected/8416ba37-dd88-46c2-bbb1-cb649aeab244-kube-api-access-7896t\") pod \"8416ba37-dd88-46c2-bbb1-cb649aeab244\" (UID: \"8416ba37-dd88-46c2-bbb1-cb649aeab244\") "
Sep 09 05:54:31.965418 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:31.965260    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8416ba37-dd88-46c2-bbb1-cb649aeab244-util\") pod \"8416ba37-dd88-46c2-bbb1-cb649aeab244\" (UID: \"8416ba37-dd88-46c2-bbb1-cb649aeab244\") "
Sep 09 05:54:31.965418 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:31.965288    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8416ba37-dd88-46c2-bbb1-cb649aeab244-bundle\") pod \"8416ba37-dd88-46c2-bbb1-cb649aeab244\" (UID: \"8416ba37-dd88-46c2-bbb1-cb649aeab244\") "
Sep 09 05:54:31.966064 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:31.966009    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8416ba37-dd88-46c2-bbb1-cb649aeab244-bundle" (OuterVolumeSpecName: "bundle") pod "8416ba37-dd88-46c2-bbb1-cb649aeab244" (UID: "8416ba37-dd88-46c2-bbb1-cb649aeab244"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:54:31.967317 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:31.967294    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8416ba37-dd88-46c2-bbb1-cb649aeab244-kube-api-access-7896t" (OuterVolumeSpecName: "kube-api-access-7896t") pod "8416ba37-dd88-46c2-bbb1-cb649aeab244" (UID: "8416ba37-dd88-46c2-bbb1-cb649aeab244"). InnerVolumeSpecName "kube-api-access-7896t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:54:31.970788 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:31.970763    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8416ba37-dd88-46c2-bbb1-cb649aeab244-util" (OuterVolumeSpecName: "util") pod "8416ba37-dd88-46c2-bbb1-cb649aeab244" (UID: "8416ba37-dd88-46c2-bbb1-cb649aeab244"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:54:32.066328 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:32.066295    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8416ba37-dd88-46c2-bbb1-cb649aeab244-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:54:32.066328 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:32.066323    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8416ba37-dd88-46c2-bbb1-cb649aeab244-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:54:32.066328 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:32.066335    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7896t\" (UniqueName: \"kubernetes.io/projected/8416ba37-dd88-46c2-bbb1-cb649aeab244-kube-api-access-7896t\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:54:32.722149 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:32.722117    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4"
Sep 09 05:54:32.722324 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:32.722139    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p8d4" event={"ID":"8416ba37-dd88-46c2-bbb1-cb649aeab244","Type":"ContainerDied","Data":"e212e55c2b303e12fc03bb1415a4e2ce98afa4fc42387383a25c0cacd9b91430"}
Sep 09 05:54:32.722324 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:32.722199    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e212e55c2b303e12fc03bb1415a4e2ce98afa4fc42387383a25c0cacd9b91430"
Sep 09 05:54:33.207443 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:33.207412    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 05:54:33.209766 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:33.209738    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 05:54:33.226014 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:33.225990    2564 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 05:54:42.080882 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.080853    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"]
Sep 09 05:54:42.121521 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.081159    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8416ba37-dd88-46c2-bbb1-cb649aeab244" containerName="util"
Sep 09 05:54:42.121521 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.081169    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8416ba37-dd88-46c2-bbb1-cb649aeab244" containerName="util"
Sep 09 05:54:42.121521 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.081184    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8416ba37-dd88-46c2-bbb1-cb649aeab244" containerName="extract"
Sep 09 05:54:42.121521 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.081189    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8416ba37-dd88-46c2-bbb1-cb649aeab244" containerName="extract"
Sep 09 05:54:42.121521 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.081197    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8416ba37-dd88-46c2-bbb1-cb649aeab244" containerName="pull"
Sep 09 05:54:42.121521 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.081202    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8416ba37-dd88-46c2-bbb1-cb649aeab244" containerName="pull"
Sep 09 05:54:42.121521 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.081250    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8416ba37-dd88-46c2-bbb1-cb649aeab244" containerName="extract"
Sep 09 05:54:42.126815 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.126786    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"]
Sep 09 05:54:42.126923 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.126885    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"
Sep 09 05:54:42.129253 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.129234    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 05:54:42.130172 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.130155    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xrhv9\""
Sep 09 05:54:42.130277 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.130159    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 05:54:42.247275 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.247235    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86721d24-77c8-4c96-8237-1bc50e6da523-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn\" (UID: \"86721d24-77c8-4c96-8237-1bc50e6da523\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"
Sep 09 05:54:42.247465 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.247323    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86721d24-77c8-4c96-8237-1bc50e6da523-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn\" (UID: \"86721d24-77c8-4c96-8237-1bc50e6da523\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"
Sep 09 05:54:42.247465 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.247351    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dcgv\" (UniqueName: \"kubernetes.io/projected/86721d24-77c8-4c96-8237-1bc50e6da523-kube-api-access-5dcgv\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn\" (UID: \"86721d24-77c8-4c96-8237-1bc50e6da523\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"
Sep 09 05:54:42.301421 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.301391    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"]
Sep 09 05:54:42.323553 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.323527    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.326562 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.326534    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xfnz9\""
Sep 09 05:54:42.326667 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.326559    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 05:54:42.326667 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.326572    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 05:54:42.326667 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.326619    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 05:54:42.326793 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.326779    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 05:54:42.327200 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.327182    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"]
Sep 09 05:54:42.348749 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.348674    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dcgv\" (UniqueName: \"kubernetes.io/projected/86721d24-77c8-4c96-8237-1bc50e6da523-kube-api-access-5dcgv\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn\" (UID: \"86721d24-77c8-4c96-8237-1bc50e6da523\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"
Sep 09 05:54:42.348882 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.348745    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86721d24-77c8-4c96-8237-1bc50e6da523-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn\" (UID: \"86721d24-77c8-4c96-8237-1bc50e6da523\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"
Sep 09 05:54:42.348882 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.348832    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86721d24-77c8-4c96-8237-1bc50e6da523-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn\" (UID: \"86721d24-77c8-4c96-8237-1bc50e6da523\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"
Sep 09 05:54:42.349128 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.349108    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86721d24-77c8-4c96-8237-1bc50e6da523-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn\" (UID: \"86721d24-77c8-4c96-8237-1bc50e6da523\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"
Sep 09 05:54:42.349201 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.349165    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86721d24-77c8-4c96-8237-1bc50e6da523-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn\" (UID: \"86721d24-77c8-4c96-8237-1bc50e6da523\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"
Sep 09 05:54:42.357435 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.357411    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dcgv\" (UniqueName: \"kubernetes.io/projected/86721d24-77c8-4c96-8237-1bc50e6da523-kube-api-access-5dcgv\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn\" (UID: \"86721d24-77c8-4c96-8237-1bc50e6da523\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"
Sep 09 05:54:42.436037 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.436009    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"
Sep 09 05:54:42.449867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.449837    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6193bc5-a652-497f-b53f-8fa768a17fcc-tmp\") pod \"opendatahub-operator-controller-manager-76b47fccc7-cjspw\" (UID: \"b6193bc5-a652-497f-b53f-8fa768a17fcc\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.450002 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.449916    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6193bc5-a652-497f-b53f-8fa768a17fcc-manifests\") pod \"opendatahub-operator-controller-manager-76b47fccc7-cjspw\" (UID: \"b6193bc5-a652-497f-b53f-8fa768a17fcc\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.450002 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.449956    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6193bc5-a652-497f-b53f-8fa768a17fcc-webhook-cert\") pod \"opendatahub-operator-controller-manager-76b47fccc7-cjspw\" (UID: \"b6193bc5-a652-497f-b53f-8fa768a17fcc\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.450002 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.449987    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7skns\" (UniqueName: \"kubernetes.io/projected/b6193bc5-a652-497f-b53f-8fa768a17fcc-kube-api-access-7skns\") pod \"opendatahub-operator-controller-manager-76b47fccc7-cjspw\" (UID: \"b6193bc5-a652-497f-b53f-8fa768a17fcc\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.450162 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.450091    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6193bc5-a652-497f-b53f-8fa768a17fcc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76b47fccc7-cjspw\" (UID: \"b6193bc5-a652-497f-b53f-8fa768a17fcc\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.551366 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.551331    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6193bc5-a652-497f-b53f-8fa768a17fcc-webhook-cert\") pod \"opendatahub-operator-controller-manager-76b47fccc7-cjspw\" (UID: \"b6193bc5-a652-497f-b53f-8fa768a17fcc\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.551366 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.551371    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7skns\" (UniqueName: \"kubernetes.io/projected/b6193bc5-a652-497f-b53f-8fa768a17fcc-kube-api-access-7skns\") pod \"opendatahub-operator-controller-manager-76b47fccc7-cjspw\" (UID: \"b6193bc5-a652-497f-b53f-8fa768a17fcc\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.551594 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.551401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6193bc5-a652-497f-b53f-8fa768a17fcc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76b47fccc7-cjspw\" (UID: \"b6193bc5-a652-497f-b53f-8fa768a17fcc\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.551594 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.551442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6193bc5-a652-497f-b53f-8fa768a17fcc-tmp\") pod \"opendatahub-operator-controller-manager-76b47fccc7-cjspw\" (UID: \"b6193bc5-a652-497f-b53f-8fa768a17fcc\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.551594 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.551483    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6193bc5-a652-497f-b53f-8fa768a17fcc-manifests\") pod \"opendatahub-operator-controller-manager-76b47fccc7-cjspw\" (UID: \"b6193bc5-a652-497f-b53f-8fa768a17fcc\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.551902 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.551878    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6193bc5-a652-497f-b53f-8fa768a17fcc-manifests\") pod \"opendatahub-operator-controller-manager-76b47fccc7-cjspw\" (UID: \"b6193bc5-a652-497f-b53f-8fa768a17fcc\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.553717 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.553697    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6193bc5-a652-497f-b53f-8fa768a17fcc-tmp\") pod \"opendatahub-operator-controller-manager-76b47fccc7-cjspw\" (UID: \"b6193bc5-a652-497f-b53f-8fa768a17fcc\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.554052 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.554032    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6193bc5-a652-497f-b53f-8fa768a17fcc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76b47fccc7-cjspw\" (UID: \"b6193bc5-a652-497f-b53f-8fa768a17fcc\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.554123 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.554033    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6193bc5-a652-497f-b53f-8fa768a17fcc-webhook-cert\") pod \"opendatahub-operator-controller-manager-76b47fccc7-cjspw\" (UID: \"b6193bc5-a652-497f-b53f-8fa768a17fcc\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.559746 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.559720    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7skns\" (UniqueName: \"kubernetes.io/projected/b6193bc5-a652-497f-b53f-8fa768a17fcc-kube-api-access-7skns\") pod \"opendatahub-operator-controller-manager-76b47fccc7-cjspw\" (UID: \"b6193bc5-a652-497f-b53f-8fa768a17fcc\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.635297 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.635216    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:42.712165 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.711817    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"]
Sep 09 05:54:42.773128 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.773099    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn" event={"ID":"86721d24-77c8-4c96-8237-1bc50e6da523","Type":"ContainerStarted","Data":"d17af457b5b5cc893d4f6f4e8cb17959594fab55379b4d2a56d79d946ad4b8bb"}
Sep 09 05:54:42.773223 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.773153    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn" event={"ID":"86721d24-77c8-4c96-8237-1bc50e6da523","Type":"ContainerStarted","Data":"7433bfdd92443df19de03d271b99c1632caebfd977f27f91dd8c365f694e5cf7"}
Sep 09 05:54:42.924331 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.924305    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"]
Sep 09 05:54:42.926858 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:54:42.926825    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6193bc5_a652_497f_b53f_8fa768a17fcc.slice/crio-2d13d3f76d16c74592b92a2a12af824a2e1568b205cd13537586f63e44368219 WatchSource:0}: Error finding container 2d13d3f76d16c74592b92a2a12af824a2e1568b205cd13537586f63e44368219: Status 404 returned error can't find the container with id 2d13d3f76d16c74592b92a2a12af824a2e1568b205cd13537586f63e44368219
Sep 09 05:54:42.928791 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:42.928774    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 05:54:43.781810 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:43.781772    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw" event={"ID":"b6193bc5-a652-497f-b53f-8fa768a17fcc","Type":"ContainerStarted","Data":"2d13d3f76d16c74592b92a2a12af824a2e1568b205cd13537586f63e44368219"}
Sep 09 05:54:43.783968 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:43.783936    2564 generic.go:356] "Generic (PLEG): container finished" podID="86721d24-77c8-4c96-8237-1bc50e6da523" containerID="d17af457b5b5cc893d4f6f4e8cb17959594fab55379b4d2a56d79d946ad4b8bb" exitCode=0
Sep 09 05:54:43.784086 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:43.784019    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn" event={"ID":"86721d24-77c8-4c96-8237-1bc50e6da523","Type":"ContainerDied","Data":"d17af457b5b5cc893d4f6f4e8cb17959594fab55379b4d2a56d79d946ad4b8bb"}
Sep 09 05:54:46.803042 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:46.802999    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw" event={"ID":"b6193bc5-a652-497f-b53f-8fa768a17fcc","Type":"ContainerStarted","Data":"be29cceb55a43d57efed09ca1780475e449025733e35bd855461203a1334be2b"}
Sep 09 05:54:46.804991 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:46.804966    2564 generic.go:356] "Generic (PLEG): container finished" podID="86721d24-77c8-4c96-8237-1bc50e6da523" containerID="7915ef5bcf9b7eca23856e85adb71b87b42fab6e125e1934b28e597d911705a9" exitCode=0
Sep 09 05:54:46.805102 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:46.805045    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn" event={"ID":"86721d24-77c8-4c96-8237-1bc50e6da523","Type":"ContainerDied","Data":"7915ef5bcf9b7eca23856e85adb71b87b42fab6e125e1934b28e597d911705a9"}
Sep 09 05:54:47.812383 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:47.812347    2564 generic.go:356] "Generic (PLEG): container finished" podID="b6193bc5-a652-497f-b53f-8fa768a17fcc" containerID="be29cceb55a43d57efed09ca1780475e449025733e35bd855461203a1334be2b" exitCode=0
Sep 09 05:54:47.812827 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:47.812441    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw" event={"ID":"b6193bc5-a652-497f-b53f-8fa768a17fcc","Type":"ContainerDied","Data":"be29cceb55a43d57efed09ca1780475e449025733e35bd855461203a1334be2b"}
Sep 09 05:54:47.814733 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:47.814658    2564 generic.go:356] "Generic (PLEG): container finished" podID="86721d24-77c8-4c96-8237-1bc50e6da523" containerID="2e7473e4038adbe8d2552d0cb5e1d23fb1233b9c1ed1192d41a6f1e3d06ba835" exitCode=0
Sep 09 05:54:47.814733 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:47.814688    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn" event={"ID":"86721d24-77c8-4c96-8237-1bc50e6da523","Type":"ContainerDied","Data":"2e7473e4038adbe8d2552d0cb5e1d23fb1233b9c1ed1192d41a6f1e3d06ba835"}
Sep 09 05:54:48.821987 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:48.821943    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw" event={"ID":"b6193bc5-a652-497f-b53f-8fa768a17fcc","Type":"ContainerStarted","Data":"a4331232f445d4061d82b74e03949737270dceac174e0e11726afc96d4b1862f"}
Sep 09 05:54:48.822416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:48.822027    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:54:48.844394 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:48.844331    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw" podStartSLOduration=2.562673422 podStartE2EDuration="6.844312873s" totalImagesPullingTime="3.203530258s" totalInitContainerRuntime="1.078109193s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:54:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:54:48.842682034 +0000 UTC m=+316.059011917" watchObservedRunningTime="2026-09-09 05:54:48.844312873 +0000 UTC m=+316.060642756"
Sep 09 05:54:48.960747 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:48.960720    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"
Sep 09 05:54:49.109690 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:49.109608    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86721d24-77c8-4c96-8237-1bc50e6da523-bundle\") pod \"86721d24-77c8-4c96-8237-1bc50e6da523\" (UID: \"86721d24-77c8-4c96-8237-1bc50e6da523\") "
Sep 09 05:54:49.109690 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:49.109656    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86721d24-77c8-4c96-8237-1bc50e6da523-util\") pod \"86721d24-77c8-4c96-8237-1bc50e6da523\" (UID: \"86721d24-77c8-4c96-8237-1bc50e6da523\") "
Sep 09 05:54:49.109690 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:49.109692    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dcgv\" (UniqueName: \"kubernetes.io/projected/86721d24-77c8-4c96-8237-1bc50e6da523-kube-api-access-5dcgv\") pod \"86721d24-77c8-4c96-8237-1bc50e6da523\" (UID: \"86721d24-77c8-4c96-8237-1bc50e6da523\") "
Sep 09 05:54:49.110391 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:49.110353    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86721d24-77c8-4c96-8237-1bc50e6da523-bundle" (OuterVolumeSpecName: "bundle") pod "86721d24-77c8-4c96-8237-1bc50e6da523" (UID: "86721d24-77c8-4c96-8237-1bc50e6da523"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:54:49.111822 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:49.111789    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86721d24-77c8-4c96-8237-1bc50e6da523-kube-api-access-5dcgv" (OuterVolumeSpecName: "kube-api-access-5dcgv") pod "86721d24-77c8-4c96-8237-1bc50e6da523" (UID: "86721d24-77c8-4c96-8237-1bc50e6da523"). InnerVolumeSpecName "kube-api-access-5dcgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:54:49.115395 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:49.115366    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86721d24-77c8-4c96-8237-1bc50e6da523-util" (OuterVolumeSpecName: "util") pod "86721d24-77c8-4c96-8237-1bc50e6da523" (UID: "86721d24-77c8-4c96-8237-1bc50e6da523"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:54:49.210376 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:49.210346    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86721d24-77c8-4c96-8237-1bc50e6da523-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:54:49.210376 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:49.210372    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dcgv\" (UniqueName: \"kubernetes.io/projected/86721d24-77c8-4c96-8237-1bc50e6da523-kube-api-access-5dcgv\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:54:49.210595 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:49.210387    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86721d24-77c8-4c96-8237-1bc50e6da523-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:54:49.828711 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:49.828681    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn"
Sep 09 05:54:49.829107 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:49.828724    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gb4xn" event={"ID":"86721d24-77c8-4c96-8237-1bc50e6da523","Type":"ContainerDied","Data":"7433bfdd92443df19de03d271b99c1632caebfd977f27f91dd8c365f694e5cf7"}
Sep 09 05:54:49.829107 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:49.828767    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7433bfdd92443df19de03d271b99c1632caebfd977f27f91dd8c365f694e5cf7"
Sep 09 05:54:59.831811 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:54:59.831781    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-cjspw"
Sep 09 05:55:01.759079 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.759047    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf"]
Sep 09 05:55:01.761994 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.759890    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86721d24-77c8-4c96-8237-1bc50e6da523" containerName="util"
Sep 09 05:55:01.761994 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.759934    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="86721d24-77c8-4c96-8237-1bc50e6da523" containerName="util"
Sep 09 05:55:01.761994 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.759964    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86721d24-77c8-4c96-8237-1bc50e6da523" containerName="extract"
Sep 09 05:55:01.761994 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.759972    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="86721d24-77c8-4c96-8237-1bc50e6da523" containerName="extract"
Sep 09 05:55:01.761994 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.760007    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86721d24-77c8-4c96-8237-1bc50e6da523" containerName="pull"
Sep 09 05:55:01.761994 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.760015    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="86721d24-77c8-4c96-8237-1bc50e6da523" containerName="pull"
Sep 09 05:55:01.761994 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.760254    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="86721d24-77c8-4c96-8237-1bc50e6da523" containerName="extract"
Sep 09 05:55:01.766821 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.766794    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf"
Sep 09 05:55:01.769503 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.769395    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 05:55:01.769503 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.769409    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 05:55:01.770539 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.770515    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zv7bh\""
Sep 09 05:55:01.772233 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.772211    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf"]
Sep 09 05:55:01.922669 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.922633    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lskbb\" (UniqueName: \"kubernetes.io/projected/85a77e83-90c0-41b3-8f32-7e2013c06f2a-kube-api-access-lskbb\") pod \"kube-auth-proxy-7f4d448d7f-hj2jf\" (UID: \"85a77e83-90c0-41b3-8f32-7e2013c06f2a\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf"
Sep 09 05:55:01.922853 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.922684    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85a77e83-90c0-41b3-8f32-7e2013c06f2a-tmp\") pod \"kube-auth-proxy-7f4d448d7f-hj2jf\" (UID: \"85a77e83-90c0-41b3-8f32-7e2013c06f2a\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf"
Sep 09 05:55:01.922853 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:01.922709    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85a77e83-90c0-41b3-8f32-7e2013c06f2a-tls-certs\") pod \"kube-auth-proxy-7f4d448d7f-hj2jf\" (UID: \"85a77e83-90c0-41b3-8f32-7e2013c06f2a\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf"
Sep 09 05:55:02.023876 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:02.023797    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85a77e83-90c0-41b3-8f32-7e2013c06f2a-tmp\") pod \"kube-auth-proxy-7f4d448d7f-hj2jf\" (UID: \"85a77e83-90c0-41b3-8f32-7e2013c06f2a\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf"
Sep 09 05:55:02.023876 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:02.023834    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85a77e83-90c0-41b3-8f32-7e2013c06f2a-tls-certs\") pod \"kube-auth-proxy-7f4d448d7f-hj2jf\" (UID: \"85a77e83-90c0-41b3-8f32-7e2013c06f2a\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf"
Sep 09 05:55:02.024055 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:02.023892    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lskbb\" (UniqueName: \"kubernetes.io/projected/85a77e83-90c0-41b3-8f32-7e2013c06f2a-kube-api-access-lskbb\") pod \"kube-auth-proxy-7f4d448d7f-hj2jf\" (UID: \"85a77e83-90c0-41b3-8f32-7e2013c06f2a\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf"
Sep 09 05:55:02.026101 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:02.026070    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85a77e83-90c0-41b3-8f32-7e2013c06f2a-tmp\") pod \"kube-auth-proxy-7f4d448d7f-hj2jf\" (UID: \"85a77e83-90c0-41b3-8f32-7e2013c06f2a\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf"
Sep 09 05:55:02.026422 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:02.026398    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85a77e83-90c0-41b3-8f32-7e2013c06f2a-tls-certs\") pod \"kube-auth-proxy-7f4d448d7f-hj2jf\" (UID: \"85a77e83-90c0-41b3-8f32-7e2013c06f2a\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf"
Sep 09 05:55:02.032009 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:02.031978    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lskbb\" (UniqueName: \"kubernetes.io/projected/85a77e83-90c0-41b3-8f32-7e2013c06f2a-kube-api-access-lskbb\") pod \"kube-auth-proxy-7f4d448d7f-hj2jf\" (UID: \"85a77e83-90c0-41b3-8f32-7e2013c06f2a\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf"
Sep 09 05:55:02.077589 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:02.077555    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf"
Sep 09 05:55:02.350867 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:02.348577    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf"]
Sep 09 05:55:02.352942 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:55:02.352911    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a77e83_90c0_41b3_8f32_7e2013c06f2a.slice/crio-bace3add1144d5e29e427b865ddfb2e92ac5ee33d861c78efb3884d6fe9a46fb WatchSource:0}: Error finding container bace3add1144d5e29e427b865ddfb2e92ac5ee33d861c78efb3884d6fe9a46fb: Status 404 returned error can't find the container with id bace3add1144d5e29e427b865ddfb2e92ac5ee33d861c78efb3884d6fe9a46fb
Sep 09 05:55:02.890580 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:02.890544    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf" event={"ID":"85a77e83-90c0-41b3-8f32-7e2013c06f2a","Type":"ContainerStarted","Data":"bace3add1144d5e29e427b865ddfb2e92ac5ee33d861c78efb3884d6fe9a46fb"}
Sep 09 05:55:05.910313 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:05.910278    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf" event={"ID":"85a77e83-90c0-41b3-8f32-7e2013c06f2a","Type":"ContainerStarted","Data":"bd6a9a11c40a21173d20220a3564733a97ca9344fc0480b98bdc6fc83c3d4356"}
Sep 09 05:55:05.926021 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:05.925974    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-hj2jf" podStartSLOduration=2.220833164 podStartE2EDuration="4.925959362s" totalImagesPullingTime="2.705126198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:55:05.924916655 +0000 UTC m=+333.141246540" watchObservedRunningTime="2026-09-09 05:55:05.925959362 +0000 UTC m=+333.142289245"
Sep 09 05:55:09.072705 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.072672    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"]
Sep 09 05:55:09.076261 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.076235    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.078722 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.078691    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-62l27\""
Sep 09 05:55:09.078722 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.078703    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 05:55:09.082982 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.082663    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3616474-7c24-43cf-bfea-ee196983f08f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.082982 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.082721    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3616474-7c24-43cf-bfea-ee196983f08f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.082982 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.082820    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3616474-7c24-43cf-bfea-ee196983f08f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.082982 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.082875    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3616474-7c24-43cf-bfea-ee196983f08f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.082982 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.082935    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3616474-7c24-43cf-bfea-ee196983f08f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.082982 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.082963    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbhnz\" (UniqueName: \"kubernetes.io/projected/d3616474-7c24-43cf-bfea-ee196983f08f-kube-api-access-zbhnz\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.083296 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.082997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3616474-7c24-43cf-bfea-ee196983f08f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.083296 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.083109    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3616474-7c24-43cf-bfea-ee196983f08f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.083296 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.083153    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3616474-7c24-43cf-bfea-ee196983f08f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.087074 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.087049    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"]
Sep 09 05:55:09.183940 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.183906    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3616474-7c24-43cf-bfea-ee196983f08f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.183940 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.183943    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3616474-7c24-43cf-bfea-ee196983f08f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.184164 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.183962    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3616474-7c24-43cf-bfea-ee196983f08f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.184164 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.183982    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3616474-7c24-43cf-bfea-ee196983f08f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.184164 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.184010    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbhnz\" (UniqueName: \"kubernetes.io/projected/d3616474-7c24-43cf-bfea-ee196983f08f-kube-api-access-zbhnz\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.184164 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.184041    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3616474-7c24-43cf-bfea-ee196983f08f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.184164 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.184104    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3616474-7c24-43cf-bfea-ee196983f08f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.184164 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.184135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3616474-7c24-43cf-bfea-ee196983f08f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.184478 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.184171    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3616474-7c24-43cf-bfea-ee196983f08f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.184478 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.184386    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3616474-7c24-43cf-bfea-ee196983f08f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.184621 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.184479    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3616474-7c24-43cf-bfea-ee196983f08f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.184621 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.184593    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3616474-7c24-43cf-bfea-ee196983f08f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.184621 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.184614    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3616474-7c24-43cf-bfea-ee196983f08f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.184856 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.184838    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3616474-7c24-43cf-bfea-ee196983f08f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.186259 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.186235    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3616474-7c24-43cf-bfea-ee196983f08f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.186554 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.186538    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3616474-7c24-43cf-bfea-ee196983f08f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.192259 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.192234    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3616474-7c24-43cf-bfea-ee196983f08f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.192374 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.192273    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbhnz\" (UniqueName: \"kubernetes.io/projected/d3616474-7c24-43cf-bfea-ee196983f08f-kube-api-access-zbhnz\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs\" (UID: \"d3616474-7c24-43cf-bfea-ee196983f08f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.387718 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.387631    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:09.662766 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.662652    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"]
Sep 09 05:55:09.665447 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:55:09.665420    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3616474_7c24_43cf_bfea_ee196983f08f.slice/crio-03a68e5adee48af2d8382acba7a258a47156bf45dada8d2f7fe8b6320002a0fd WatchSource:0}: Error finding container 03a68e5adee48af2d8382acba7a258a47156bf45dada8d2f7fe8b6320002a0fd: Status 404 returned error can't find the container with id 03a68e5adee48af2d8382acba7a258a47156bf45dada8d2f7fe8b6320002a0fd
Sep 09 05:55:09.935857 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:09.935775    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs" event={"ID":"d3616474-7c24-43cf-bfea-ee196983f08f","Type":"ContainerStarted","Data":"03a68e5adee48af2d8382acba7a258a47156bf45dada8d2f7fe8b6320002a0fd"}
Sep 09 05:55:12.377659 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:12.377624    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gck77"]
Sep 09 05:55:12.384650 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:12.384565    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gck77"
Sep 09 05:55:12.387456 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:12.387425    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 09 05:55:12.388015 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:12.387990    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9qfgn\""
Sep 09 05:55:12.409438 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:12.409366    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gck77"]
Sep 09 05:55:12.411652 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:12.411615    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6c1cf4c-1991-4869-b663-781a5cc7fbc0-cert\") pod \"odh-model-controller-7998c67dc4-gck77\" (UID: \"c6c1cf4c-1991-4869-b663-781a5cc7fbc0\") " pod="opendatahub/odh-model-controller-7998c67dc4-gck77"
Sep 09 05:55:12.411770 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:12.411714    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8txv4\" (UniqueName: \"kubernetes.io/projected/c6c1cf4c-1991-4869-b663-781a5cc7fbc0-kube-api-access-8txv4\") pod \"odh-model-controller-7998c67dc4-gck77\" (UID: \"c6c1cf4c-1991-4869-b663-781a5cc7fbc0\") " pod="opendatahub/odh-model-controller-7998c67dc4-gck77"
Sep 09 05:55:12.512937 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:12.512902    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6c1cf4c-1991-4869-b663-781a5cc7fbc0-cert\") pod \"odh-model-controller-7998c67dc4-gck77\" (UID: \"c6c1cf4c-1991-4869-b663-781a5cc7fbc0\") " pod="opendatahub/odh-model-controller-7998c67dc4-gck77"
Sep 09 05:55:12.513099 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:12.512947    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8txv4\" (UniqueName: \"kubernetes.io/projected/c6c1cf4c-1991-4869-b663-781a5cc7fbc0-kube-api-access-8txv4\") pod \"odh-model-controller-7998c67dc4-gck77\" (UID: \"c6c1cf4c-1991-4869-b663-781a5cc7fbc0\") " pod="opendatahub/odh-model-controller-7998c67dc4-gck77"
Sep 09 05:55:12.513099 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:55:12.513060    2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 09 05:55:12.513218 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:55:12.513134    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c1cf4c-1991-4869-b663-781a5cc7fbc0-cert podName:c6c1cf4c-1991-4869-b663-781a5cc7fbc0 nodeName:}" failed. No retries permitted until 2026-09-09 05:55:13.013113036 +0000 UTC m=+340.229442935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c6c1cf4c-1991-4869-b663-781a5cc7fbc0-cert") pod "odh-model-controller-7998c67dc4-gck77" (UID: "c6c1cf4c-1991-4869-b663-781a5cc7fbc0") : secret "odh-model-controller-webhook-cert" not found
Sep 09 05:55:12.521597 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:12.521574    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8txv4\" (UniqueName: \"kubernetes.io/projected/c6c1cf4c-1991-4869-b663-781a5cc7fbc0-kube-api-access-8txv4\") pod \"odh-model-controller-7998c67dc4-gck77\" (UID: \"c6c1cf4c-1991-4869-b663-781a5cc7fbc0\") " pod="opendatahub/odh-model-controller-7998c67dc4-gck77"
Sep 09 05:55:12.687961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:12.687913    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 05:55:12.688054 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:12.688007    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 05:55:12.688096 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:12.688048    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 05:55:12.956764 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:12.956673    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs" event={"ID":"d3616474-7c24-43cf-bfea-ee196983f08f","Type":"ContainerStarted","Data":"c95ed24b84ffeae1fcc9ca0ce038e752a304cbe5a21047215821408a66766760"}
Sep 09 05:55:12.975961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:12.975911    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs" podStartSLOduration=0.956046609 podStartE2EDuration="3.97589607s" totalImagesPullingTime="3.019849461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:55:12.974918572 +0000 UTC m=+340.191248468" watchObservedRunningTime="2026-09-09 05:55:12.97589607 +0000 UTC m=+340.192225953"
Sep 09 05:55:13.017552 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:13.017517    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6c1cf4c-1991-4869-b663-781a5cc7fbc0-cert\") pod \"odh-model-controller-7998c67dc4-gck77\" (UID: \"c6c1cf4c-1991-4869-b663-781a5cc7fbc0\") " pod="opendatahub/odh-model-controller-7998c67dc4-gck77"
Sep 09 05:55:13.020010 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:13.019980    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6c1cf4c-1991-4869-b663-781a5cc7fbc0-cert\") pod \"odh-model-controller-7998c67dc4-gck77\" (UID: \"c6c1cf4c-1991-4869-b663-781a5cc7fbc0\") " pod="opendatahub/odh-model-controller-7998c67dc4-gck77"
Sep 09 05:55:13.298477 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:13.298396    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gck77"
Sep 09 05:55:13.388871 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:13.388836    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:13.393882 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:13.393859    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:13.567369 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:13.567343    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gck77"]
Sep 09 05:55:13.569242 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:55:13.569213    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c1cf4c_1991_4869_b663_781a5cc7fbc0.slice/crio-7eb2a685ae9ef59b85031ab359a9d31ff4a352c5d97f53daee182bb4adaee025 WatchSource:0}: Error finding container 7eb2a685ae9ef59b85031ab359a9d31ff4a352c5d97f53daee182bb4adaee025: Status 404 returned error can't find the container with id 7eb2a685ae9ef59b85031ab359a9d31ff4a352c5d97f53daee182bb4adaee025
Sep 09 05:55:13.963040 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:13.963007    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gck77" event={"ID":"c6c1cf4c-1991-4869-b663-781a5cc7fbc0","Type":"ContainerStarted","Data":"7eb2a685ae9ef59b85031ab359a9d31ff4a352c5d97f53daee182bb4adaee025"}
Sep 09 05:55:13.963235 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:13.963214    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:13.964277 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:13.964256    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs"
Sep 09 05:55:16.981538 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:16.981477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gck77" event={"ID":"c6c1cf4c-1991-4869-b663-781a5cc7fbc0","Type":"ContainerStarted","Data":"8860147f51875f9e83f12ac1803c9011a02b1e7c722e063ddf2dd489821a349d"}
Sep 09 05:55:16.981993 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:16.981600    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-gck77"
Sep 09 05:55:16.998180 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:16.998138    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-gck77" podStartSLOduration=2.10561513 podStartE2EDuration="4.998124839s" totalImagesPullingTime="2.892509709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:55:16.996275749 +0000 UTC m=+344.212605632" watchObservedRunningTime="2026-09-09 05:55:16.998124839 +0000 UTC m=+344.214454723"
Sep 09 05:55:19.093532 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.093475    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"]
Sep 09 05:55:19.099376 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.099331    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.101946 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.101919    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-njpwm\""
Sep 09 05:55:19.102143 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.102119    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 05:55:19.102261 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.102150    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 05:55:19.102339 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.102325    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 05:55:19.111848 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.111826    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"]
Sep 09 05:55:19.174289 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.174254    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/edf95dbf-0662-4437-9a94-95c165b05069-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.174443 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.174294    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf95dbf-0662-4437-9a94-95c165b05069-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.174443 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.174323    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/edf95dbf-0662-4437-9a94-95c165b05069-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.174443 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.174414    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmnhd\" (UniqueName: \"kubernetes.io/projected/edf95dbf-0662-4437-9a94-95c165b05069-kube-api-access-nmnhd\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.174588 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.174455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/edf95dbf-0662-4437-9a94-95c165b05069-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.174588 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.174511    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf95dbf-0662-4437-9a94-95c165b05069-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.174588 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.174541    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/edf95dbf-0662-4437-9a94-95c165b05069-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.275369 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.275338    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf95dbf-0662-4437-9a94-95c165b05069-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.275546 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.275377    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/edf95dbf-0662-4437-9a94-95c165b05069-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.275546 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.275410    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmnhd\" (UniqueName: \"kubernetes.io/projected/edf95dbf-0662-4437-9a94-95c165b05069-kube-api-access-nmnhd\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.275546 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.275434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/edf95dbf-0662-4437-9a94-95c165b05069-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.275546 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.275460    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf95dbf-0662-4437-9a94-95c165b05069-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.275546 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.275485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/edf95dbf-0662-4437-9a94-95c165b05069-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.275815 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.275548    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/edf95dbf-0662-4437-9a94-95c165b05069-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.276208 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.276171    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/edf95dbf-0662-4437-9a94-95c165b05069-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.277826 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.277799    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf95dbf-0662-4437-9a94-95c165b05069-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.278036 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.278016    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/edf95dbf-0662-4437-9a94-95c165b05069-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.278118 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.278100    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/edf95dbf-0662-4437-9a94-95c165b05069-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.278155 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.278102    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/edf95dbf-0662-4437-9a94-95c165b05069-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.286631 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.286607    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf95dbf-0662-4437-9a94-95c165b05069-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.287004 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.286984    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmnhd\" (UniqueName: \"kubernetes.io/projected/edf95dbf-0662-4437-9a94-95c165b05069-kube-api-access-nmnhd\") pod \"istiod-openshift-gateway-5677ccb49b-4mscg\" (UID: \"edf95dbf-0662-4437-9a94-95c165b05069\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.411180 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.411094    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:19.704479 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.704450    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"]
Sep 09 05:55:19.706483 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:55:19.706443    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf95dbf_0662_4437_9a94_95c165b05069.slice/crio-17982e7709fecbea4ed22f903c771a7a0093e4a79eb74e1682c7187d07fa75f9 WatchSource:0}: Error finding container 17982e7709fecbea4ed22f903c771a7a0093e4a79eb74e1682c7187d07fa75f9: Status 404 returned error can't find the container with id 17982e7709fecbea4ed22f903c771a7a0093e4a79eb74e1682c7187d07fa75f9
Sep 09 05:55:19.997350 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:19.997273    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg" event={"ID":"edf95dbf-0662-4437-9a94-95c165b05069","Type":"ContainerStarted","Data":"17982e7709fecbea4ed22f903c771a7a0093e4a79eb74e1682c7187d07fa75f9"}
Sep 09 05:55:21.921208 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:21.921170    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 05:55:21.921608 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:21.921252    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 05:55:22.012782 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:22.012741    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg" event={"ID":"edf95dbf-0662-4437-9a94-95c165b05069","Type":"ContainerStarted","Data":"57f16c11d3a170d754bd25f21fa513d52882372968f3887bc3852fc6e0f488ec"}
Sep 09 05:55:22.012964 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:22.012813    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:22.014335 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:22.014310    2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-4mscg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:8080/ready\": dial tcp 10.132.0.40:8080: connect: connection refused" start-of-body=
Sep 09 05:55:22.014451 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:22.014356    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg" podUID="edf95dbf-0662-4437-9a94-95c165b05069" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.40:8080/ready\": dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 05:55:22.039567 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:22.039480    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg" podStartSLOduration=0.827358331 podStartE2EDuration="3.039456187s" totalImagesPullingTime="2.212097856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:55:22.037603874 +0000 UTC m=+349.253933824" watchObservedRunningTime="2026-09-09 05:55:22.039456187 +0000 UTC m=+349.255786072"
Sep 09 05:55:23.019481 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:23.019434    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mscg"
Sep 09 05:55:27.988364 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:27.988288    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-gck77"
Sep 09 05:55:47.834632 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:47.834598    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54f94d85f7-dqg4q"]
Sep 09 05:55:47.848979 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:47.848947    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54f94d85f7-dqg4q"]
Sep 09 05:55:47.849137 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:47.849069    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.038622 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.038583    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f86d8f73-4276-413e-9890-686f31835873-console-config\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.038795 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.038636    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f86d8f73-4276-413e-9890-686f31835873-service-ca\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.038795 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.038660    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f86d8f73-4276-413e-9890-686f31835873-console-serving-cert\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.038795 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.038681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f86d8f73-4276-413e-9890-686f31835873-trusted-ca-bundle\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.038795 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.038699    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f86d8f73-4276-413e-9890-686f31835873-oauth-serving-cert\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.038795 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.038715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f86d8f73-4276-413e-9890-686f31835873-console-oauth-config\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.039066 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.038830    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp846\" (UniqueName: \"kubernetes.io/projected/f86d8f73-4276-413e-9890-686f31835873-kube-api-access-jp846\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.139359 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.139273    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f86d8f73-4276-413e-9890-686f31835873-console-config\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.139359 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.139319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f86d8f73-4276-413e-9890-686f31835873-service-ca\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.139359 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.139351    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f86d8f73-4276-413e-9890-686f31835873-console-serving-cert\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.139657 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.139378    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f86d8f73-4276-413e-9890-686f31835873-trusted-ca-bundle\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.139657 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.139402    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f86d8f73-4276-413e-9890-686f31835873-oauth-serving-cert\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.139657 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.139430    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f86d8f73-4276-413e-9890-686f31835873-console-oauth-config\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.139657 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.139462    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp846\" (UniqueName: \"kubernetes.io/projected/f86d8f73-4276-413e-9890-686f31835873-kube-api-access-jp846\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.140164 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.140135    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f86d8f73-4276-413e-9890-686f31835873-console-config\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.140384 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.140360    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f86d8f73-4276-413e-9890-686f31835873-trusted-ca-bundle\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.140384 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.140373    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f86d8f73-4276-413e-9890-686f31835873-service-ca\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.140570 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.140399    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f86d8f73-4276-413e-9890-686f31835873-oauth-serving-cert\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.142020 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.141993    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f86d8f73-4276-413e-9890-686f31835873-console-oauth-config\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.142102 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.142063    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f86d8f73-4276-413e-9890-686f31835873-console-serving-cert\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.147543 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.147521    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp846\" (UniqueName: \"kubernetes.io/projected/f86d8f73-4276-413e-9890-686f31835873-kube-api-access-jp846\") pod \"console-54f94d85f7-dqg4q\" (UID: \"f86d8f73-4276-413e-9890-686f31835873\") " pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.159236 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.159208    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:48.463392 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:48.463370    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54f94d85f7-dqg4q"]
Sep 09 05:55:48.465294 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:55:48.465268    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86d8f73_4276_413e_9890_686f31835873.slice/crio-13d14d14d2cc7c93191e78c5fe8a78d5cfaffd3c643abfc95966d122f477f5e5 WatchSource:0}: Error finding container 13d14d14d2cc7c93191e78c5fe8a78d5cfaffd3c643abfc95966d122f477f5e5: Status 404 returned error can't find the container with id 13d14d14d2cc7c93191e78c5fe8a78d5cfaffd3c643abfc95966d122f477f5e5
Sep 09 05:55:49.147245 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:49.147209    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f94d85f7-dqg4q" event={"ID":"f86d8f73-4276-413e-9890-686f31835873","Type":"ContainerStarted","Data":"dec3b110071584e6d58b25a6f123838631da958687b23078c44c95992046fcbf"}
Sep 09 05:55:49.147765 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:49.147261    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f94d85f7-dqg4q" event={"ID":"f86d8f73-4276-413e-9890-686f31835873","Type":"ContainerStarted","Data":"13d14d14d2cc7c93191e78c5fe8a78d5cfaffd3c643abfc95966d122f477f5e5"}
Sep 09 05:55:49.166668 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:49.166620    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54f94d85f7-dqg4q" podStartSLOduration=2.166605559 podStartE2EDuration="2.166605559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:55:49.163951802 +0000 UTC m=+376.380281687" watchObservedRunningTime="2026-09-09 05:55:49.166605559 +0000 UTC m=+376.382935442"
Sep 09 05:55:58.160295 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:58.160259    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:58.160700 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:58.160323    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:58.165127 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:58.165102    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:58.194577 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:58.194551    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54f94d85f7-dqg4q"
Sep 09 05:55:58.241072 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:55:58.241027    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59c7bb8db9-hdr9s"]
Sep 09 05:56:17.014510 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.014469    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"]
Sep 09 05:56:17.018412 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.018382    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"
Sep 09 05:56:17.021054 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.021032    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 05:56:17.022013 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.021994    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xrhv9\""
Sep 09 05:56:17.022073 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.022049    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 05:56:17.027619 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.027594    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"]
Sep 09 05:56:17.093550 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.093515    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn\" (UID: \"1d74d353-4d8b-4ccc-b35a-cfc7193f7711\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"
Sep 09 05:56:17.093550 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.093555    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn\" (UID: \"1d74d353-4d8b-4ccc-b35a-cfc7193f7711\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"
Sep 09 05:56:17.093742 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.093576    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v5lp\" (UniqueName: \"kubernetes.io/projected/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-kube-api-access-2v5lp\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn\" (UID: \"1d74d353-4d8b-4ccc-b35a-cfc7193f7711\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"
Sep 09 05:56:17.115200 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.115171    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"]
Sep 09 05:56:17.118679 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.118663    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"
Sep 09 05:56:17.126850 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.126827    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"]
Sep 09 05:56:17.194665 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.194637    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn\" (UID: \"1d74d353-4d8b-4ccc-b35a-cfc7193f7711\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"
Sep 09 05:56:17.194814 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.194681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f52ba47-8829-4295-8af5-f17ee034e656-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt\" (UID: \"9f52ba47-8829-4295-8af5-f17ee034e656\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"
Sep 09 05:56:17.194814 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.194705    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn\" (UID: \"1d74d353-4d8b-4ccc-b35a-cfc7193f7711\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"
Sep 09 05:56:17.194814 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.194770    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v5lp\" (UniqueName: \"kubernetes.io/projected/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-kube-api-access-2v5lp\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn\" (UID: \"1d74d353-4d8b-4ccc-b35a-cfc7193f7711\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"
Sep 09 05:56:17.195004 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.194833    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh9f8\" (UniqueName: \"kubernetes.io/projected/9f52ba47-8829-4295-8af5-f17ee034e656-kube-api-access-lh9f8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt\" (UID: \"9f52ba47-8829-4295-8af5-f17ee034e656\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"
Sep 09 05:56:17.195004 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.194856    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f52ba47-8829-4295-8af5-f17ee034e656-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt\" (UID: \"9f52ba47-8829-4295-8af5-f17ee034e656\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"
Sep 09 05:56:17.195112 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.195080    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn\" (UID: \"1d74d353-4d8b-4ccc-b35a-cfc7193f7711\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"
Sep 09 05:56:17.195159 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.195084    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn\" (UID: \"1d74d353-4d8b-4ccc-b35a-cfc7193f7711\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"
Sep 09 05:56:17.205553 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.205534    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v5lp\" (UniqueName: \"kubernetes.io/projected/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-kube-api-access-2v5lp\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn\" (UID: \"1d74d353-4d8b-4ccc-b35a-cfc7193f7711\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"
Sep 09 05:56:17.213472 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.213450    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"]
Sep 09 05:56:17.217050 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.217035    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"
Sep 09 05:56:17.227999 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.227977    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"]
Sep 09 05:56:17.295734 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.295643    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb417471-50b6-4ee9-8377-0c1b0c7585d5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk\" (UID: \"bb417471-50b6-4ee9-8377-0c1b0c7585d5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"
Sep 09 05:56:17.295734 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.295699    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f52ba47-8829-4295-8af5-f17ee034e656-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt\" (UID: \"9f52ba47-8829-4295-8af5-f17ee034e656\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"
Sep 09 05:56:17.295961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.295775    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh9f8\" (UniqueName: \"kubernetes.io/projected/9f52ba47-8829-4295-8af5-f17ee034e656-kube-api-access-lh9f8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt\" (UID: \"9f52ba47-8829-4295-8af5-f17ee034e656\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"
Sep 09 05:56:17.295961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.295805    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb417471-50b6-4ee9-8377-0c1b0c7585d5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk\" (UID: \"bb417471-50b6-4ee9-8377-0c1b0c7585d5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"
Sep 09 05:56:17.295961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.295831    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f52ba47-8829-4295-8af5-f17ee034e656-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt\" (UID: \"9f52ba47-8829-4295-8af5-f17ee034e656\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"
Sep 09 05:56:17.295961 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.295857    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdd5d\" (UniqueName: \"kubernetes.io/projected/bb417471-50b6-4ee9-8377-0c1b0c7585d5-kube-api-access-gdd5d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk\" (UID: \"bb417471-50b6-4ee9-8377-0c1b0c7585d5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"
Sep 09 05:56:17.296112 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.296084    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f52ba47-8829-4295-8af5-f17ee034e656-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt\" (UID: \"9f52ba47-8829-4295-8af5-f17ee034e656\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"
Sep 09 05:56:17.296150 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.296109    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f52ba47-8829-4295-8af5-f17ee034e656-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt\" (UID: \"9f52ba47-8829-4295-8af5-f17ee034e656\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"
Sep 09 05:56:17.303726 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.303695    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh9f8\" (UniqueName: \"kubernetes.io/projected/9f52ba47-8829-4295-8af5-f17ee034e656-kube-api-access-lh9f8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt\" (UID: \"9f52ba47-8829-4295-8af5-f17ee034e656\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"
Sep 09 05:56:17.318163 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.318141    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"]
Sep 09 05:56:17.321896 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.321882    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"
Sep 09 05:56:17.329309 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.329128    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"
Sep 09 05:56:17.330344 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.330323    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"]
Sep 09 05:56:17.397116 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.397091    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb417471-50b6-4ee9-8377-0c1b0c7585d5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk\" (UID: \"bb417471-50b6-4ee9-8377-0c1b0c7585d5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"
Sep 09 05:56:17.397237 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.397125    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ce78476-c8d1-410c-9362-0069a22e6824-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9\" (UID: \"8ce78476-c8d1-410c-9362-0069a22e6824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"
Sep 09 05:56:17.397237 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.397145    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ce78476-c8d1-410c-9362-0069a22e6824-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9\" (UID: \"8ce78476-c8d1-410c-9362-0069a22e6824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"
Sep 09 05:56:17.397323 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.397247    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbxhg\" (UniqueName: \"kubernetes.io/projected/8ce78476-c8d1-410c-9362-0069a22e6824-kube-api-access-lbxhg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9\" (UID: \"8ce78476-c8d1-410c-9362-0069a22e6824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"
Sep 09 05:56:17.397323 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.397306    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb417471-50b6-4ee9-8377-0c1b0c7585d5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk\" (UID: \"bb417471-50b6-4ee9-8377-0c1b0c7585d5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"
Sep 09 05:56:17.397392 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.397339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdd5d\" (UniqueName: \"kubernetes.io/projected/bb417471-50b6-4ee9-8377-0c1b0c7585d5-kube-api-access-gdd5d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk\" (UID: \"bb417471-50b6-4ee9-8377-0c1b0c7585d5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"
Sep 09 05:56:17.397603 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.397583    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb417471-50b6-4ee9-8377-0c1b0c7585d5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk\" (UID: \"bb417471-50b6-4ee9-8377-0c1b0c7585d5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"
Sep 09 05:56:17.397659 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.397630    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb417471-50b6-4ee9-8377-0c1b0c7585d5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk\" (UID: \"bb417471-50b6-4ee9-8377-0c1b0c7585d5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"
Sep 09 05:56:17.406035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.406007    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdd5d\" (UniqueName: \"kubernetes.io/projected/bb417471-50b6-4ee9-8377-0c1b0c7585d5-kube-api-access-gdd5d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk\" (UID: \"bb417471-50b6-4ee9-8377-0c1b0c7585d5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"
Sep 09 05:56:17.428851 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.428830    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"
Sep 09 05:56:17.498251 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.498214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ce78476-c8d1-410c-9362-0069a22e6824-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9\" (UID: \"8ce78476-c8d1-410c-9362-0069a22e6824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"
Sep 09 05:56:17.498379 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.498268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ce78476-c8d1-410c-9362-0069a22e6824-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9\" (UID: \"8ce78476-c8d1-410c-9362-0069a22e6824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"
Sep 09 05:56:17.498379 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.498340    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbxhg\" (UniqueName: \"kubernetes.io/projected/8ce78476-c8d1-410c-9362-0069a22e6824-kube-api-access-lbxhg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9\" (UID: \"8ce78476-c8d1-410c-9362-0069a22e6824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"
Sep 09 05:56:17.498665 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.498639    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ce78476-c8d1-410c-9362-0069a22e6824-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9\" (UID: \"8ce78476-c8d1-410c-9362-0069a22e6824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"
Sep 09 05:56:17.498755 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.498661    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ce78476-c8d1-410c-9362-0069a22e6824-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9\" (UID: \"8ce78476-c8d1-410c-9362-0069a22e6824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"
Sep 09 05:56:17.506170 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.506148    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbxhg\" (UniqueName: \"kubernetes.io/projected/8ce78476-c8d1-410c-9362-0069a22e6824-kube-api-access-lbxhg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9\" (UID: \"8ce78476-c8d1-410c-9362-0069a22e6824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"
Sep 09 05:56:17.525976 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.525951    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"
Sep 09 05:56:17.602838 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.602774    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"]
Sep 09 05:56:17.605463 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:56:17.605438    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d74d353_4d8b_4ccc_b35a_cfc7193f7711.slice/crio-d30359668312f0880c7c593d9ad595da85efcbb488205e32175311fa5493876c WatchSource:0}: Error finding container d30359668312f0880c7c593d9ad595da85efcbb488205e32175311fa5493876c: Status 404 returned error can't find the container with id d30359668312f0880c7c593d9ad595da85efcbb488205e32175311fa5493876c
Sep 09 05:56:17.632105 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.632082    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"
Sep 09 05:56:17.704318 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.704251    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"]
Sep 09 05:56:17.708958 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:56:17.708925    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f52ba47_8829_4295_8af5_f17ee034e656.slice/crio-309db444538f8f05e84241850067b1b150d94c252f94bcdfe402ec9ed6d4069d WatchSource:0}: Error finding container 309db444538f8f05e84241850067b1b150d94c252f94bcdfe402ec9ed6d4069d: Status 404 returned error can't find the container with id 309db444538f8f05e84241850067b1b150d94c252f94bcdfe402ec9ed6d4069d
Sep 09 05:56:17.798652 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.798630    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"]
Sep 09 05:56:17.809000 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:56:17.808973    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb417471_50b6_4ee9_8377_0c1b0c7585d5.slice/crio-5fd39018d69eccd45c1ae69c2a0854cfea6720ec3bb24d0d5070de39065bd2f5 WatchSource:0}: Error finding container 5fd39018d69eccd45c1ae69c2a0854cfea6720ec3bb24d0d5070de39065bd2f5: Status 404 returned error can't find the container with id 5fd39018d69eccd45c1ae69c2a0854cfea6720ec3bb24d0d5070de39065bd2f5
Sep 09 05:56:17.904244 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:17.904220    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"]
Sep 09 05:56:17.973342 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:56:17.973306    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce78476_c8d1_410c_9362_0069a22e6824.slice/crio-ad9bee7ec4c69a8234fbd88d51a88910b542c725f98cd062257605a944992c47 WatchSource:0}: Error finding container ad9bee7ec4c69a8234fbd88d51a88910b542c725f98cd062257605a944992c47: Status 404 returned error can't find the container with id ad9bee7ec4c69a8234fbd88d51a88910b542c725f98cd062257605a944992c47
Sep 09 05:56:18.290003 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:18.289910    2564 generic.go:356] "Generic (PLEG): container finished" podID="9f52ba47-8829-4295-8af5-f17ee034e656" containerID="4c3cd6493ccc4650c3247e283e4e8f0c93eac19fdc7cc20222e6040f019f9013" exitCode=0
Sep 09 05:56:18.290426 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:18.289999    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt" event={"ID":"9f52ba47-8829-4295-8af5-f17ee034e656","Type":"ContainerDied","Data":"4c3cd6493ccc4650c3247e283e4e8f0c93eac19fdc7cc20222e6040f019f9013"}
Sep 09 05:56:18.290426 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:18.290086    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt" event={"ID":"9f52ba47-8829-4295-8af5-f17ee034e656","Type":"ContainerStarted","Data":"309db444538f8f05e84241850067b1b150d94c252f94bcdfe402ec9ed6d4069d"}
Sep 09 05:56:18.291793 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:18.291608    2564 generic.go:356] "Generic (PLEG): container finished" podID="8ce78476-c8d1-410c-9362-0069a22e6824" containerID="3e7bdc009c1e6a309d84189fa0d443190f03f2bc062b4dddee4c760f7e8ac342" exitCode=0
Sep 09 05:56:18.291793 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:18.291675    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9" event={"ID":"8ce78476-c8d1-410c-9362-0069a22e6824","Type":"ContainerDied","Data":"3e7bdc009c1e6a309d84189fa0d443190f03f2bc062b4dddee4c760f7e8ac342"}
Sep 09 05:56:18.291793 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:18.291716    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9" event={"ID":"8ce78476-c8d1-410c-9362-0069a22e6824","Type":"ContainerStarted","Data":"ad9bee7ec4c69a8234fbd88d51a88910b542c725f98cd062257605a944992c47"}
Sep 09 05:56:18.293322 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:18.293300    2564 generic.go:356] "Generic (PLEG): container finished" podID="1d74d353-4d8b-4ccc-b35a-cfc7193f7711" containerID="c72039ea368122cf4561a06b48fddd37191b176f187f2f984645c1e2f729aabe" exitCode=0
Sep 09 05:56:18.293417 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:18.293379    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn" event={"ID":"1d74d353-4d8b-4ccc-b35a-cfc7193f7711","Type":"ContainerDied","Data":"c72039ea368122cf4561a06b48fddd37191b176f187f2f984645c1e2f729aabe"}
Sep 09 05:56:18.293478 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:18.293438    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn" event={"ID":"1d74d353-4d8b-4ccc-b35a-cfc7193f7711","Type":"ContainerStarted","Data":"d30359668312f0880c7c593d9ad595da85efcbb488205e32175311fa5493876c"}
Sep 09 05:56:18.295048 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:18.295023    2564 generic.go:356] "Generic (PLEG): container finished" podID="bb417471-50b6-4ee9-8377-0c1b0c7585d5" containerID="34a2896b0a393ff83736ddec07c0ef2d63ac79255f8e7a6dc0055b588c55eeeb" exitCode=0
Sep 09 05:56:18.295134 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:18.295060    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk" event={"ID":"bb417471-50b6-4ee9-8377-0c1b0c7585d5","Type":"ContainerDied","Data":"34a2896b0a393ff83736ddec07c0ef2d63ac79255f8e7a6dc0055b588c55eeeb"}
Sep 09 05:56:18.295134 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:18.295093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk" event={"ID":"bb417471-50b6-4ee9-8377-0c1b0c7585d5","Type":"ContainerStarted","Data":"5fd39018d69eccd45c1ae69c2a0854cfea6720ec3bb24d0d5070de39065bd2f5"}
Sep 09 05:56:20.309423 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:20.309391    2564 generic.go:356] "Generic (PLEG): container finished" podID="9f52ba47-8829-4295-8af5-f17ee034e656" containerID="244ad26973808de62e7579cf49b221c8efb72a0133b127815b31896ecd795d9e" exitCode=0
Sep 09 05:56:20.309854 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:20.309454    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt" event={"ID":"9f52ba47-8829-4295-8af5-f17ee034e656","Type":"ContainerDied","Data":"244ad26973808de62e7579cf49b221c8efb72a0133b127815b31896ecd795d9e"}
Sep 09 05:56:20.311541 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:20.311518    2564 generic.go:356] "Generic (PLEG): container finished" podID="8ce78476-c8d1-410c-9362-0069a22e6824" containerID="2aa566d6960b8745bd01002ef74c9c517bf0cfa7601290ebc0ca41dc222e4d76" exitCode=0
Sep 09 05:56:20.311626 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:20.311598    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9" event={"ID":"8ce78476-c8d1-410c-9362-0069a22e6824","Type":"ContainerDied","Data":"2aa566d6960b8745bd01002ef74c9c517bf0cfa7601290ebc0ca41dc222e4d76"}
Sep 09 05:56:20.313685 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:20.313667    2564 generic.go:356] "Generic (PLEG): container finished" podID="1d74d353-4d8b-4ccc-b35a-cfc7193f7711" containerID="999757ec0481d03aa0514c6226d58b850bc0e62b8f5852d1e5823363e498875e" exitCode=0
Sep 09 05:56:20.313769 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:20.313724    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn" event={"ID":"1d74d353-4d8b-4ccc-b35a-cfc7193f7711","Type":"ContainerDied","Data":"999757ec0481d03aa0514c6226d58b850bc0e62b8f5852d1e5823363e498875e"}
Sep 09 05:56:21.320751 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:21.320722    2564 generic.go:356] "Generic (PLEG): container finished" podID="8ce78476-c8d1-410c-9362-0069a22e6824" containerID="c0168070e48676e140bfb01dcce638e9b1dd6796bbeb20f04fd921c1fd233e3f" exitCode=0
Sep 09 05:56:21.321130 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:21.320797    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9" event={"ID":"8ce78476-c8d1-410c-9362-0069a22e6824","Type":"ContainerDied","Data":"c0168070e48676e140bfb01dcce638e9b1dd6796bbeb20f04fd921c1fd233e3f"}
Sep 09 05:56:21.322876 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:21.322857    2564 generic.go:356] "Generic (PLEG): container finished" podID="1d74d353-4d8b-4ccc-b35a-cfc7193f7711" containerID="df69901e8825bd473e5f177fffe2a2055c216e02bc0890fc5473e34d75119112" exitCode=0
Sep 09 05:56:21.322971 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:21.322941    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn" event={"ID":"1d74d353-4d8b-4ccc-b35a-cfc7193f7711","Type":"ContainerDied","Data":"df69901e8825bd473e5f177fffe2a2055c216e02bc0890fc5473e34d75119112"}
Sep 09 05:56:21.324714 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:21.324691    2564 generic.go:356] "Generic (PLEG): container finished" podID="bb417471-50b6-4ee9-8377-0c1b0c7585d5" containerID="a9aba916f9335cafadcbf5b3b9c01fc088bb113f1e2fd90497fe4168bbf913d2" exitCode=0
Sep 09 05:56:21.324803 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:21.324764    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk" event={"ID":"bb417471-50b6-4ee9-8377-0c1b0c7585d5","Type":"ContainerDied","Data":"a9aba916f9335cafadcbf5b3b9c01fc088bb113f1e2fd90497fe4168bbf913d2"}
Sep 09 05:56:21.326981 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:21.326960    2564 generic.go:356] "Generic (PLEG): container finished" podID="9f52ba47-8829-4295-8af5-f17ee034e656" containerID="e2bf61dc59ddcf35e89a06bd17544b2863e52b1d9cb7b6b10b9212016bd5af43" exitCode=0
Sep 09 05:56:21.327061 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:21.326987    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt" event={"ID":"9f52ba47-8829-4295-8af5-f17ee034e656","Type":"ContainerDied","Data":"e2bf61dc59ddcf35e89a06bd17544b2863e52b1d9cb7b6b10b9212016bd5af43"}
Sep 09 05:56:22.334297 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.334262    2564 generic.go:356] "Generic (PLEG): container finished" podID="bb417471-50b6-4ee9-8377-0c1b0c7585d5" containerID="bc332facda915d6068388e17de1fafc7d1d17a583f1c5ef4eefd2861f646f458" exitCode=0
Sep 09 05:56:22.334737 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.334369    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk" event={"ID":"bb417471-50b6-4ee9-8377-0c1b0c7585d5","Type":"ContainerDied","Data":"bc332facda915d6068388e17de1fafc7d1d17a583f1c5ef4eefd2861f646f458"}
Sep 09 05:56:22.477121 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.477095    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"
Sep 09 05:56:22.522385 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.522362    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"
Sep 09 05:56:22.525183 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.525163    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"
Sep 09 05:56:22.542847 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.542825    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ce78476-c8d1-410c-9362-0069a22e6824-util\") pod \"8ce78476-c8d1-410c-9362-0069a22e6824\" (UID: \"8ce78476-c8d1-410c-9362-0069a22e6824\") "
Sep 09 05:56:22.542959 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.542925    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbxhg\" (UniqueName: \"kubernetes.io/projected/8ce78476-c8d1-410c-9362-0069a22e6824-kube-api-access-lbxhg\") pod \"8ce78476-c8d1-410c-9362-0069a22e6824\" (UID: \"8ce78476-c8d1-410c-9362-0069a22e6824\") "
Sep 09 05:56:22.542959 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.542951    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ce78476-c8d1-410c-9362-0069a22e6824-bundle\") pod \"8ce78476-c8d1-410c-9362-0069a22e6824\" (UID: \"8ce78476-c8d1-410c-9362-0069a22e6824\") "
Sep 09 05:56:22.543685 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.543655    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce78476-c8d1-410c-9362-0069a22e6824-bundle" (OuterVolumeSpecName: "bundle") pod "8ce78476-c8d1-410c-9362-0069a22e6824" (UID: "8ce78476-c8d1-410c-9362-0069a22e6824"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:56:22.545357 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.545322    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce78476-c8d1-410c-9362-0069a22e6824-kube-api-access-lbxhg" (OuterVolumeSpecName: "kube-api-access-lbxhg") pod "8ce78476-c8d1-410c-9362-0069a22e6824" (UID: "8ce78476-c8d1-410c-9362-0069a22e6824"). InnerVolumeSpecName "kube-api-access-lbxhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:56:22.548353 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.548328    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce78476-c8d1-410c-9362-0069a22e6824-util" (OuterVolumeSpecName: "util") pod "8ce78476-c8d1-410c-9362-0069a22e6824" (UID: "8ce78476-c8d1-410c-9362-0069a22e6824"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:56:22.643687 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.643607    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-bundle\") pod \"1d74d353-4d8b-4ccc-b35a-cfc7193f7711\" (UID: \"1d74d353-4d8b-4ccc-b35a-cfc7193f7711\") "
Sep 09 05:56:22.643687 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.643675    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-util\") pod \"1d74d353-4d8b-4ccc-b35a-cfc7193f7711\" (UID: \"1d74d353-4d8b-4ccc-b35a-cfc7193f7711\") "
Sep 09 05:56:22.643861 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.643703    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f52ba47-8829-4295-8af5-f17ee034e656-util\") pod \"9f52ba47-8829-4295-8af5-f17ee034e656\" (UID: \"9f52ba47-8829-4295-8af5-f17ee034e656\") "
Sep 09 05:56:22.643861 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.643758    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh9f8\" (UniqueName: \"kubernetes.io/projected/9f52ba47-8829-4295-8af5-f17ee034e656-kube-api-access-lh9f8\") pod \"9f52ba47-8829-4295-8af5-f17ee034e656\" (UID: \"9f52ba47-8829-4295-8af5-f17ee034e656\") "
Sep 09 05:56:22.643861 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.643779    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v5lp\" (UniqueName: \"kubernetes.io/projected/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-kube-api-access-2v5lp\") pod \"1d74d353-4d8b-4ccc-b35a-cfc7193f7711\" (UID: \"1d74d353-4d8b-4ccc-b35a-cfc7193f7711\") "
Sep 09 05:56:22.643861 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.643806    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f52ba47-8829-4295-8af5-f17ee034e656-bundle\") pod \"9f52ba47-8829-4295-8af5-f17ee034e656\" (UID: \"9f52ba47-8829-4295-8af5-f17ee034e656\") "
Sep 09 05:56:22.644042 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.644027    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ce78476-c8d1-410c-9362-0069a22e6824-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:22.644100 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.644047    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbxhg\" (UniqueName: \"kubernetes.io/projected/8ce78476-c8d1-410c-9362-0069a22e6824-kube-api-access-lbxhg\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:22.644100 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.644062    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ce78476-c8d1-410c-9362-0069a22e6824-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:22.644396 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.644365    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f52ba47-8829-4295-8af5-f17ee034e656-bundle" (OuterVolumeSpecName: "bundle") pod "9f52ba47-8829-4295-8af5-f17ee034e656" (UID: "9f52ba47-8829-4295-8af5-f17ee034e656"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:56:22.644533 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.644505    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-bundle" (OuterVolumeSpecName: "bundle") pod "1d74d353-4d8b-4ccc-b35a-cfc7193f7711" (UID: "1d74d353-4d8b-4ccc-b35a-cfc7193f7711"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:56:22.645865 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.645842    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f52ba47-8829-4295-8af5-f17ee034e656-kube-api-access-lh9f8" (OuterVolumeSpecName: "kube-api-access-lh9f8") pod "9f52ba47-8829-4295-8af5-f17ee034e656" (UID: "9f52ba47-8829-4295-8af5-f17ee034e656"). InnerVolumeSpecName "kube-api-access-lh9f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:56:22.646229 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.646209    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-kube-api-access-2v5lp" (OuterVolumeSpecName: "kube-api-access-2v5lp") pod "1d74d353-4d8b-4ccc-b35a-cfc7193f7711" (UID: "1d74d353-4d8b-4ccc-b35a-cfc7193f7711"). InnerVolumeSpecName "kube-api-access-2v5lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:56:22.649705 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.649685    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-util" (OuterVolumeSpecName: "util") pod "1d74d353-4d8b-4ccc-b35a-cfc7193f7711" (UID: "1d74d353-4d8b-4ccc-b35a-cfc7193f7711"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:56:22.649874 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.649853    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f52ba47-8829-4295-8af5-f17ee034e656-util" (OuterVolumeSpecName: "util") pod "9f52ba47-8829-4295-8af5-f17ee034e656" (UID: "9f52ba47-8829-4295-8af5-f17ee034e656"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:56:22.744545 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.744503    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:22.744545 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.744533    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f52ba47-8829-4295-8af5-f17ee034e656-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:22.744766 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.744571    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh9f8\" (UniqueName: \"kubernetes.io/projected/9f52ba47-8829-4295-8af5-f17ee034e656-kube-api-access-lh9f8\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:22.744766 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.744585    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v5lp\" (UniqueName: \"kubernetes.io/projected/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-kube-api-access-2v5lp\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:22.744766 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.744599    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f52ba47-8829-4295-8af5-f17ee034e656-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:22.744766 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:22.744610    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d74d353-4d8b-4ccc-b35a-cfc7193f7711-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:23.265693 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.265648    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59c7bb8db9-hdr9s" podUID="57aed479-fc13-4425-99e0-83815dd3172c" containerName="console" containerID="cri-o://14c5a5062287d9c310fd657ad9c69fed9365bdb05d7c2970a9f67ce541787bd7" gracePeriod=15
Sep 09 05:56:23.342116 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.342080    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9" event={"ID":"8ce78476-c8d1-410c-9362-0069a22e6824","Type":"ContainerDied","Data":"ad9bee7ec4c69a8234fbd88d51a88910b542c725f98cd062257605a944992c47"}
Sep 09 05:56:23.342445 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.342137    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad9bee7ec4c69a8234fbd88d51a88910b542c725f98cd062257605a944992c47"
Sep 09 05:56:23.342445 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.342095    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xw4l9"
Sep 09 05:56:23.344267 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.344234    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn" event={"ID":"1d74d353-4d8b-4ccc-b35a-cfc7193f7711","Type":"ContainerDied","Data":"d30359668312f0880c7c593d9ad595da85efcbb488205e32175311fa5493876c"}
Sep 09 05:56:23.344368 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.344288    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d30359668312f0880c7c593d9ad595da85efcbb488205e32175311fa5493876c"
Sep 09 05:56:23.344368 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.344249    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as6ksn"
Sep 09 05:56:23.346289 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.346266    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt"
Sep 09 05:56:23.346380 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.346301    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69cbdt" event={"ID":"9f52ba47-8829-4295-8af5-f17ee034e656","Type":"ContainerDied","Data":"309db444538f8f05e84241850067b1b150d94c252f94bcdfe402ec9ed6d4069d"}
Sep 09 05:56:23.346380 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.346347    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="309db444538f8f05e84241850067b1b150d94c252f94bcdfe402ec9ed6d4069d"
Sep 09 05:56:23.492372 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.492347    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"
Sep 09 05:56:23.531427 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.531376    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:56:23.652045 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.652016    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-service-ca\") pod \"57aed479-fc13-4425-99e0-83815dd3172c\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") "
Sep 09 05:56:23.652236 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.652062    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57aed479-fc13-4425-99e0-83815dd3172c-console-serving-cert\") pod \"57aed479-fc13-4425-99e0-83815dd3172c\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") "
Sep 09 05:56:23.652236 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.652091    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb417471-50b6-4ee9-8377-0c1b0c7585d5-bundle\") pod \"bb417471-50b6-4ee9-8377-0c1b0c7585d5\" (UID: \"bb417471-50b6-4ee9-8377-0c1b0c7585d5\") "
Sep 09 05:56:23.652236 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.652113    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-console-config\") pod \"57aed479-fc13-4425-99e0-83815dd3172c\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") "
Sep 09 05:56:23.652236 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.652138    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtkfm\" (UniqueName: \"kubernetes.io/projected/57aed479-fc13-4425-99e0-83815dd3172c-kube-api-access-dtkfm\") pod \"57aed479-fc13-4425-99e0-83815dd3172c\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") "
Sep 09 05:56:23.652236 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.652181    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57aed479-fc13-4425-99e0-83815dd3172c-console-oauth-config\") pod \"57aed479-fc13-4425-99e0-83815dd3172c\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") "
Sep 09 05:56:23.652236 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.652220    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-oauth-serving-cert\") pod \"57aed479-fc13-4425-99e0-83815dd3172c\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") "
Sep 09 05:56:23.652651 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.652320    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-trusted-ca-bundle\") pod \"57aed479-fc13-4425-99e0-83815dd3172c\" (UID: \"57aed479-fc13-4425-99e0-83815dd3172c\") "
Sep 09 05:56:23.652651 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.652366    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb417471-50b6-4ee9-8377-0c1b0c7585d5-util\") pod \"bb417471-50b6-4ee9-8377-0c1b0c7585d5\" (UID: \"bb417471-50b6-4ee9-8377-0c1b0c7585d5\") "
Sep 09 05:56:23.652651 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.652398    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-service-ca" (OuterVolumeSpecName: "service-ca") pod "57aed479-fc13-4425-99e0-83815dd3172c" (UID: "57aed479-fc13-4425-99e0-83815dd3172c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:56:23.652651 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.652420    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdd5d\" (UniqueName: \"kubernetes.io/projected/bb417471-50b6-4ee9-8377-0c1b0c7585d5-kube-api-access-gdd5d\") pod \"bb417471-50b6-4ee9-8377-0c1b0c7585d5\" (UID: \"bb417471-50b6-4ee9-8377-0c1b0c7585d5\") "
Sep 09 05:56:23.652651 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.652615    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-console-config" (OuterVolumeSpecName: "console-config") pod "57aed479-fc13-4425-99e0-83815dd3172c" (UID: "57aed479-fc13-4425-99e0-83815dd3172c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:56:23.653271 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.652758    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:23.653271 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.652782    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-console-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:23.653271 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.652758    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb417471-50b6-4ee9-8377-0c1b0c7585d5-bundle" (OuterVolumeSpecName: "bundle") pod "bb417471-50b6-4ee9-8377-0c1b0c7585d5" (UID: "bb417471-50b6-4ee9-8377-0c1b0c7585d5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:56:23.653271 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.653022    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "57aed479-fc13-4425-99e0-83815dd3172c" (UID: "57aed479-fc13-4425-99e0-83815dd3172c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:56:23.653506 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.653384    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "57aed479-fc13-4425-99e0-83815dd3172c" (UID: "57aed479-fc13-4425-99e0-83815dd3172c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:56:23.654601 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.654569    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57aed479-fc13-4425-99e0-83815dd3172c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "57aed479-fc13-4425-99e0-83815dd3172c" (UID: "57aed479-fc13-4425-99e0-83815dd3172c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:56:23.654707 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.654634    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57aed479-fc13-4425-99e0-83815dd3172c-kube-api-access-dtkfm" (OuterVolumeSpecName: "kube-api-access-dtkfm") pod "57aed479-fc13-4425-99e0-83815dd3172c" (UID: "57aed479-fc13-4425-99e0-83815dd3172c"). InnerVolumeSpecName "kube-api-access-dtkfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:56:23.654752 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.654708    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb417471-50b6-4ee9-8377-0c1b0c7585d5-kube-api-access-gdd5d" (OuterVolumeSpecName: "kube-api-access-gdd5d") pod "bb417471-50b6-4ee9-8377-0c1b0c7585d5" (UID: "bb417471-50b6-4ee9-8377-0c1b0c7585d5"). InnerVolumeSpecName "kube-api-access-gdd5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:56:23.654752 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.654705    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57aed479-fc13-4425-99e0-83815dd3172c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "57aed479-fc13-4425-99e0-83815dd3172c" (UID: "57aed479-fc13-4425-99e0-83815dd3172c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:56:23.658108 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.658085    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb417471-50b6-4ee9-8377-0c1b0c7585d5-util" (OuterVolumeSpecName: "util") pod "bb417471-50b6-4ee9-8377-0c1b0c7585d5" (UID: "bb417471-50b6-4ee9-8377-0c1b0c7585d5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:56:23.754072 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.754048    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdd5d\" (UniqueName: \"kubernetes.io/projected/bb417471-50b6-4ee9-8377-0c1b0c7585d5-kube-api-access-gdd5d\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:23.754072 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.754069    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57aed479-fc13-4425-99e0-83815dd3172c-console-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:23.754226 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.754079    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb417471-50b6-4ee9-8377-0c1b0c7585d5-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:23.754226 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.754089    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtkfm\" (UniqueName: \"kubernetes.io/projected/57aed479-fc13-4425-99e0-83815dd3172c-kube-api-access-dtkfm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:23.754226 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.754098    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57aed479-fc13-4425-99e0-83815dd3172c-console-oauth-config\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:23.754226 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.754106    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-oauth-serving-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:23.754226 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.754115    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57aed479-fc13-4425-99e0-83815dd3172c-trusted-ca-bundle\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:23.754226 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:23.754123    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb417471-50b6-4ee9-8377-0c1b0c7585d5-util\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:24.353505 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:24.353469    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk"
Sep 09 05:56:24.353899 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:24.353512    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8s2fk" event={"ID":"bb417471-50b6-4ee9-8377-0c1b0c7585d5","Type":"ContainerDied","Data":"5fd39018d69eccd45c1ae69c2a0854cfea6720ec3bb24d0d5070de39065bd2f5"}
Sep 09 05:56:24.353899 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:24.353563    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fd39018d69eccd45c1ae69c2a0854cfea6720ec3bb24d0d5070de39065bd2f5"
Sep 09 05:56:24.354901 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:24.354877    2564 generic.go:356] "Generic (PLEG): container finished" podID="57aed479-fc13-4425-99e0-83815dd3172c" containerID="14c5a5062287d9c310fd657ad9c69fed9365bdb05d7c2970a9f67ce541787bd7" exitCode=0
Sep 09 05:56:24.355002 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:24.354908    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c7bb8db9-hdr9s"
Sep 09 05:56:24.355002 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:24.354952    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c7bb8db9-hdr9s" event={"ID":"57aed479-fc13-4425-99e0-83815dd3172c","Type":"ContainerDied","Data":"14c5a5062287d9c310fd657ad9c69fed9365bdb05d7c2970a9f67ce541787bd7"}
Sep 09 05:56:24.355002 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:24.354992    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c7bb8db9-hdr9s" event={"ID":"57aed479-fc13-4425-99e0-83815dd3172c","Type":"ContainerDied","Data":"1b3eaf21ffe7d15ecdcfb066260f1fbefd5eca16ee6eaf52952266bcd4efa87c"}
Sep 09 05:56:24.355160 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:24.355011    2564 scope.go:117] "RemoveContainer" containerID="14c5a5062287d9c310fd657ad9c69fed9365bdb05d7c2970a9f67ce541787bd7"
Sep 09 05:56:24.365040 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:24.365021    2564 scope.go:117] "RemoveContainer" containerID="14c5a5062287d9c310fd657ad9c69fed9365bdb05d7c2970a9f67ce541787bd7"
Sep 09 05:56:24.365343 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:56:24.365318    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c5a5062287d9c310fd657ad9c69fed9365bdb05d7c2970a9f67ce541787bd7\": container with ID starting with 14c5a5062287d9c310fd657ad9c69fed9365bdb05d7c2970a9f67ce541787bd7 not found: ID does not exist" containerID="14c5a5062287d9c310fd657ad9c69fed9365bdb05d7c2970a9f67ce541787bd7"
Sep 09 05:56:24.365408 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:24.365351    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c5a5062287d9c310fd657ad9c69fed9365bdb05d7c2970a9f67ce541787bd7"} err="failed to get container status \"14c5a5062287d9c310fd657ad9c69fed9365bdb05d7c2970a9f67ce541787bd7\": rpc error: code = NotFound desc = could not find container \"14c5a5062287d9c310fd657ad9c69fed9365bdb05d7c2970a9f67ce541787bd7\": container with ID starting with 14c5a5062287d9c310fd657ad9c69fed9365bdb05d7c2970a9f67ce541787bd7 not found: ID does not exist"
Sep 09 05:56:24.381302 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:24.381276    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59c7bb8db9-hdr9s"]
Sep 09 05:56:24.384948 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:24.384923    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59c7bb8db9-hdr9s"]
Sep 09 05:56:25.307320 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:25.307288    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57aed479-fc13-4425-99e0-83815dd3172c" path="/var/lib/kubelet/pods/57aed479-fc13-4425-99e0-83815dd3172c/volumes"
Sep 09 05:56:29.552937 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.552904    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-wh9vb"]
Sep 09 05:56:29.553321 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553244    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb417471-50b6-4ee9-8377-0c1b0c7585d5" containerName="pull"
Sep 09 05:56:29.553321 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553255    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb417471-50b6-4ee9-8377-0c1b0c7585d5" containerName="pull"
Sep 09 05:56:29.553321 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553266    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d74d353-4d8b-4ccc-b35a-cfc7193f7711" containerName="extract"
Sep 09 05:56:29.553321 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553271    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d74d353-4d8b-4ccc-b35a-cfc7193f7711" containerName="extract"
Sep 09 05:56:29.553321 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553279    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f52ba47-8829-4295-8af5-f17ee034e656" containerName="pull"
Sep 09 05:56:29.553321 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553285    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f52ba47-8829-4295-8af5-f17ee034e656" containerName="pull"
Sep 09 05:56:29.553321 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553293    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce78476-c8d1-410c-9362-0069a22e6824" containerName="extract"
Sep 09 05:56:29.553321 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553300    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce78476-c8d1-410c-9362-0069a22e6824" containerName="extract"
Sep 09 05:56:29.553321 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553308    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb417471-50b6-4ee9-8377-0c1b0c7585d5" containerName="util"
Sep 09 05:56:29.553321 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553313    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb417471-50b6-4ee9-8377-0c1b0c7585d5" containerName="util"
Sep 09 05:56:29.553321 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553319    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce78476-c8d1-410c-9362-0069a22e6824" containerName="pull"
Sep 09 05:56:29.553321 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553324    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce78476-c8d1-410c-9362-0069a22e6824" containerName="pull"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553331    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57aed479-fc13-4425-99e0-83815dd3172c" containerName="console"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553336    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="57aed479-fc13-4425-99e0-83815dd3172c" containerName="console"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553347    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d74d353-4d8b-4ccc-b35a-cfc7193f7711" containerName="util"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553352    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d74d353-4d8b-4ccc-b35a-cfc7193f7711" containerName="util"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553357    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f52ba47-8829-4295-8af5-f17ee034e656" containerName="extract"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553362    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f52ba47-8829-4295-8af5-f17ee034e656" containerName="extract"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553376    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f52ba47-8829-4295-8af5-f17ee034e656" containerName="util"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553380    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f52ba47-8829-4295-8af5-f17ee034e656" containerName="util"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553386    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce78476-c8d1-410c-9362-0069a22e6824" containerName="util"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553392    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce78476-c8d1-410c-9362-0069a22e6824" containerName="util"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553398    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d74d353-4d8b-4ccc-b35a-cfc7193f7711" containerName="pull"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553402    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d74d353-4d8b-4ccc-b35a-cfc7193f7711" containerName="pull"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553407    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb417471-50b6-4ee9-8377-0c1b0c7585d5" containerName="extract"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553412    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb417471-50b6-4ee9-8377-0c1b0c7585d5" containerName="extract"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553460    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f52ba47-8829-4295-8af5-f17ee034e656" containerName="extract"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553467    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="57aed479-fc13-4425-99e0-83815dd3172c" containerName="console"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553473    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d74d353-4d8b-4ccc-b35a-cfc7193f7711" containerName="extract"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553479    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb417471-50b6-4ee9-8377-0c1b0c7585d5" containerName="extract"
Sep 09 05:56:29.553750 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.553484    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce78476-c8d1-410c-9362-0069a22e6824" containerName="extract"
Sep 09 05:56:29.556442 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.556426    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-wh9vb"
Sep 09 05:56:29.559134 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.559112    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 05:56:29.559134 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.559124    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 05:56:29.559134 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.559128    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gdvgb\""
Sep 09 05:56:29.565801 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.565777    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-wh9vb"]
Sep 09 05:56:29.703848 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.703818    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx5fv\" (UniqueName: \"kubernetes.io/projected/bf928072-5a5e-42e4-9fde-0981e8b8c14d-kube-api-access-qx5fv\") pod \"authorino-operator-76b478bd78-wh9vb\" (UID: \"bf928072-5a5e-42e4-9fde-0981e8b8c14d\") " pod="kuadrant-system/authorino-operator-76b478bd78-wh9vb"
Sep 09 05:56:29.804356 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.804270    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx5fv\" (UniqueName: \"kubernetes.io/projected/bf928072-5a5e-42e4-9fde-0981e8b8c14d-kube-api-access-qx5fv\") pod \"authorino-operator-76b478bd78-wh9vb\" (UID: \"bf928072-5a5e-42e4-9fde-0981e8b8c14d\") " pod="kuadrant-system/authorino-operator-76b478bd78-wh9vb"
Sep 09 05:56:29.812444 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.812413    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx5fv\" (UniqueName: \"kubernetes.io/projected/bf928072-5a5e-42e4-9fde-0981e8b8c14d-kube-api-access-qx5fv\") pod \"authorino-operator-76b478bd78-wh9vb\" (UID: \"bf928072-5a5e-42e4-9fde-0981e8b8c14d\") " pod="kuadrant-system/authorino-operator-76b478bd78-wh9vb"
Sep 09 05:56:29.867034 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:29.867005    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-wh9vb"
Sep 09 05:56:30.137952 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:30.137929    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-wh9vb"]
Sep 09 05:56:30.139854 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:56:30.139825    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf928072_5a5e_42e4_9fde_0981e8b8c14d.slice/crio-8effe989f9435d4a62dcb81db61499b0e2e887a055efb8879d284e2eeb4e0b76 WatchSource:0}: Error finding container 8effe989f9435d4a62dcb81db61499b0e2e887a055efb8879d284e2eeb4e0b76: Status 404 returned error can't find the container with id 8effe989f9435d4a62dcb81db61499b0e2e887a055efb8879d284e2eeb4e0b76
Sep 09 05:56:30.387485 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:30.387403    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-wh9vb" event={"ID":"bf928072-5a5e-42e4-9fde-0981e8b8c14d","Type":"ContainerStarted","Data":"8effe989f9435d4a62dcb81db61499b0e2e887a055efb8879d284e2eeb4e0b76"}
Sep 09 05:56:34.150667 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:34.150635    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rzldn"]
Sep 09 05:56:34.155328 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:34.155307    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rzldn"
Sep 09 05:56:34.157748 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:34.157726    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6z8gn\""
Sep 09 05:56:34.157861 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:34.157726    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 09 05:56:34.165102 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:34.165080    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rzldn"]
Sep 09 05:56:34.242790 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:34.242755    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxzdk\" (UniqueName: \"kubernetes.io/projected/bbe0815c-476c-4eba-802a-a67324431747-kube-api-access-jxzdk\") pod \"dns-operator-controller-manager-7b84cd7c9c-rzldn\" (UID: \"bbe0815c-476c-4eba-802a-a67324431747\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rzldn"
Sep 09 05:56:34.343370 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:34.343319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxzdk\" (UniqueName: \"kubernetes.io/projected/bbe0815c-476c-4eba-802a-a67324431747-kube-api-access-jxzdk\") pod \"dns-operator-controller-manager-7b84cd7c9c-rzldn\" (UID: \"bbe0815c-476c-4eba-802a-a67324431747\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rzldn"
Sep 09 05:56:34.351238 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:34.351202    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxzdk\" (UniqueName: \"kubernetes.io/projected/bbe0815c-476c-4eba-802a-a67324431747-kube-api-access-jxzdk\") pod \"dns-operator-controller-manager-7b84cd7c9c-rzldn\" (UID: \"bbe0815c-476c-4eba-802a-a67324431747\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rzldn"
Sep 09 05:56:34.466717 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:34.466625    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rzldn"
Sep 09 05:56:34.742966 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:34.742937    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rzldn"]
Sep 09 05:56:34.743695 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:56:34.743673    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe0815c_476c_4eba_802a_a67324431747.slice/crio-1eee1b7846c890585a67cc4e5342bcb6d402ace03baa4cdab00712e275be4ef4 WatchSource:0}: Error finding container 1eee1b7846c890585a67cc4e5342bcb6d402ace03baa4cdab00712e275be4ef4: Status 404 returned error can't find the container with id 1eee1b7846c890585a67cc4e5342bcb6d402ace03baa4cdab00712e275be4ef4
Sep 09 05:56:35.424803 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:35.424761    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rzldn" event={"ID":"bbe0815c-476c-4eba-802a-a67324431747","Type":"ContainerStarted","Data":"1eee1b7846c890585a67cc4e5342bcb6d402ace03baa4cdab00712e275be4ef4"}
Sep 09 05:56:37.437823 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:37.437784    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-wh9vb" event={"ID":"bf928072-5a5e-42e4-9fde-0981e8b8c14d","Type":"ContainerStarted","Data":"f0d1cae3aaf6872617c5f57b8c87543498598965409b5757f2647b1b35bbdd6e"}
Sep 09 05:56:37.438196 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:37.437846    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-wh9vb"
Sep 09 05:56:37.454065 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:37.454017    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-wh9vb" podStartSLOduration=1.60268482 podStartE2EDuration="8.453983384s" totalImagesPullingTime="6.851298564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:56:37.452505682 +0000 UTC m=+424.668835563" watchObservedRunningTime="2026-09-09 05:56:37.453983384 +0000 UTC m=+424.670313269"
Sep 09 05:56:38.380332 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:38.380295    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"]
Sep 09 05:56:38.383858 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:38.383841    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"
Sep 09 05:56:38.387724 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:38.387700    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pm7rx\""
Sep 09 05:56:38.394028 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:38.394003    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"]
Sep 09 05:56:38.481265 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:38.481234    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qk4m\" (UniqueName: \"kubernetes.io/projected/df379a76-12ec-4c26-8dbb-f538d82d9e9f-kube-api-access-7qk4m\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lt946\" (UID: \"df379a76-12ec-4c26-8dbb-f538d82d9e9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"
Sep 09 05:56:38.481661 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:38.481270    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df379a76-12ec-4c26-8dbb-f538d82d9e9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lt946\" (UID: \"df379a76-12ec-4c26-8dbb-f538d82d9e9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"
Sep 09 05:56:38.581934 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:38.581901    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qk4m\" (UniqueName: \"kubernetes.io/projected/df379a76-12ec-4c26-8dbb-f538d82d9e9f-kube-api-access-7qk4m\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lt946\" (UID: \"df379a76-12ec-4c26-8dbb-f538d82d9e9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"
Sep 09 05:56:38.582090 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:38.581940    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df379a76-12ec-4c26-8dbb-f538d82d9e9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lt946\" (UID: \"df379a76-12ec-4c26-8dbb-f538d82d9e9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"
Sep 09 05:56:38.582403 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:38.582385    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df379a76-12ec-4c26-8dbb-f538d82d9e9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lt946\" (UID: \"df379a76-12ec-4c26-8dbb-f538d82d9e9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"
Sep 09 05:56:38.596088 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:38.596068    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qk4m\" (UniqueName: \"kubernetes.io/projected/df379a76-12ec-4c26-8dbb-f538d82d9e9f-kube-api-access-7qk4m\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lt946\" (UID: \"df379a76-12ec-4c26-8dbb-f538d82d9e9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"
Sep 09 05:56:38.695167 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:38.695075    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"
Sep 09 05:56:38.970986 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:38.970955    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"]
Sep 09 05:56:38.972273 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:56:38.972244    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf379a76_12ec_4c26_8dbb_f538d82d9e9f.slice/crio-070e0393a2c295289c4a2f93065c4dc7f9c0e9d951257081ec85907906e68fce WatchSource:0}: Error finding container 070e0393a2c295289c4a2f93065c4dc7f9c0e9d951257081ec85907906e68fce: Status 404 returned error can't find the container with id 070e0393a2c295289c4a2f93065c4dc7f9c0e9d951257081ec85907906e68fce
Sep 09 05:56:39.449116 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:39.449085    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946" event={"ID":"df379a76-12ec-4c26-8dbb-f538d82d9e9f","Type":"ContainerStarted","Data":"070e0393a2c295289c4a2f93065c4dc7f9c0e9d951257081ec85907906e68fce"}
Sep 09 05:56:41.461403 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:41.461364    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rzldn" event={"ID":"bbe0815c-476c-4eba-802a-a67324431747","Type":"ContainerStarted","Data":"2e6485b26fe5fe91c6373ef83ca60c1d8835f43c487557dd03c6d757a0877d03"}
Sep 09 05:56:41.461850 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:41.461532    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rzldn"
Sep 09 05:56:41.478482 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:41.478434    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rzldn" podStartSLOduration=1.7708808390000002 podStartE2EDuration="7.478419879s" totalImagesPullingTime="5.70753904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:56:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:56:41.476517004 +0000 UTC m=+428.692846887" watchObservedRunningTime="2026-09-09 05:56:41.478419879 +0000 UTC m=+428.694749763"
Sep 09 05:56:45.487416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:45.487379    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946" event={"ID":"df379a76-12ec-4c26-8dbb-f538d82d9e9f","Type":"ContainerStarted","Data":"8a5c22bf38da029016fa224c1830fe0d6f1a7eaabe9ecee7944959642ae4adc7"}
Sep 09 05:56:45.487829 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:45.487517    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"
Sep 09 05:56:45.510325 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:45.510272    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946" podStartSLOduration=1.5699478930000001 podStartE2EDuration="7.510259232s" totalImagesPullingTime="5.940311339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:56:45.506977655 +0000 UTC m=+432.723307540" watchObservedRunningTime="2026-09-09 05:56:45.510259232 +0000 UTC m=+432.726589117"
Sep 09 05:56:48.444478 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:48.444444    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-wh9vb"
Sep 09 05:56:52.470845 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:52.470811    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rzldn"
Sep 09 05:56:56.495830 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:56.495753    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"
Sep 09 05:56:58.611921 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.611882    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"]
Sep 09 05:56:58.612323 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.612096    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946" podUID="df379a76-12ec-4c26-8dbb-f538d82d9e9f" containerName="manager" containerID="cri-o://8a5c22bf38da029016fa224c1830fe0d6f1a7eaabe9ecee7944959642ae4adc7" gracePeriod=2
Sep 09 05:56:58.626856 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.626830    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"]
Sep 09 05:56:58.632951 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:56:58.632910    2564 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"df379a76-12ec-4c26-8dbb-f538d82d9e9f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-lt946\": pods \"kuadrant-operator-controller-manager-76f8f686bf-lt946\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"
Sep 09 05:56:58.645667 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.645633    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"]
Sep 09 05:56:58.646109 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.646092    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df379a76-12ec-4c26-8dbb-f538d82d9e9f" containerName="manager"
Sep 09 05:56:58.646202 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.646113    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="df379a76-12ec-4c26-8dbb-f538d82d9e9f" containerName="manager"
Sep 09 05:56:58.646262 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.646214    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="df379a76-12ec-4c26-8dbb-f538d82d9e9f" containerName="manager"
Sep 09 05:56:58.649404 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.649377    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"
Sep 09 05:56:58.651465 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:56:58.651431    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-lt946\" is forbidden: User \"system:node:ip-10-0-131-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-250.ec2.internal' and this object" podUID="df379a76-12ec-4c26-8dbb-f538d82d9e9f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"
Sep 09 05:56:58.664045 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.664017    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"]
Sep 09 05:56:58.749413 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.749382    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn6v2\" (UniqueName: \"kubernetes.io/projected/78dba663-7660-4282-9be7-9647a5f3446f-kube-api-access-xn6v2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d628p\" (UID: \"78dba663-7660-4282-9be7-9647a5f3446f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"
Sep 09 05:56:58.749603 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.749515    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78dba663-7660-4282-9be7-9647a5f3446f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d628p\" (UID: \"78dba663-7660-4282-9be7-9647a5f3446f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"
Sep 09 05:56:58.850658 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.850626    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78dba663-7660-4282-9be7-9647a5f3446f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d628p\" (UID: \"78dba663-7660-4282-9be7-9647a5f3446f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"
Sep 09 05:56:58.850779 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.850711    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn6v2\" (UniqueName: \"kubernetes.io/projected/78dba663-7660-4282-9be7-9647a5f3446f-kube-api-access-xn6v2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d628p\" (UID: \"78dba663-7660-4282-9be7-9647a5f3446f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"
Sep 09 05:56:58.851021 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.851002    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78dba663-7660-4282-9be7-9647a5f3446f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d628p\" (UID: \"78dba663-7660-4282-9be7-9647a5f3446f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"
Sep 09 05:56:58.854845 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.854826    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"
Sep 09 05:56:58.856964 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:56:58.856919    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-lt946\" is forbidden: User \"system:node:ip-10-0-131-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-250.ec2.internal' and this object" podUID="df379a76-12ec-4c26-8dbb-f538d82d9e9f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"
Sep 09 05:56:58.859226 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.859201    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn6v2\" (UniqueName: \"kubernetes.io/projected/78dba663-7660-4282-9be7-9647a5f3446f-kube-api-access-xn6v2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d628p\" (UID: \"78dba663-7660-4282-9be7-9647a5f3446f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"
Sep 09 05:56:58.951461 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.951373    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qk4m\" (UniqueName: \"kubernetes.io/projected/df379a76-12ec-4c26-8dbb-f538d82d9e9f-kube-api-access-7qk4m\") pod \"df379a76-12ec-4c26-8dbb-f538d82d9e9f\" (UID: \"df379a76-12ec-4c26-8dbb-f538d82d9e9f\") "
Sep 09 05:56:58.951461 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.951419    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df379a76-12ec-4c26-8dbb-f538d82d9e9f-extensions-socket-volume\") pod \"df379a76-12ec-4c26-8dbb-f538d82d9e9f\" (UID: \"df379a76-12ec-4c26-8dbb-f538d82d9e9f\") "
Sep 09 05:56:58.951926 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.951897    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df379a76-12ec-4c26-8dbb-f538d82d9e9f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "df379a76-12ec-4c26-8dbb-f538d82d9e9f" (UID: "df379a76-12ec-4c26-8dbb-f538d82d9e9f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:56:58.953593 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.953566    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df379a76-12ec-4c26-8dbb-f538d82d9e9f-kube-api-access-7qk4m" (OuterVolumeSpecName: "kube-api-access-7qk4m") pod "df379a76-12ec-4c26-8dbb-f538d82d9e9f" (UID: "df379a76-12ec-4c26-8dbb-f538d82d9e9f"). InnerVolumeSpecName "kube-api-access-7qk4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:56:58.991894 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:58.991864    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"
Sep 09 05:56:59.053462 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:59.053090    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qk4m\" (UniqueName: \"kubernetes.io/projected/df379a76-12ec-4c26-8dbb-f538d82d9e9f-kube-api-access-7qk4m\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:59.053462 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:59.053125    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df379a76-12ec-4c26-8dbb-f538d82d9e9f-extensions-socket-volume\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:56:59.274959 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:59.274932    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"]
Sep 09 05:56:59.277819 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:56:59.277790    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78dba663_7660_4282_9be7_9647a5f3446f.slice/crio-2ac7b45aae4dbaf78e6aa4195987afbae9df362ef0b0fbe49dde5a7f700a255c WatchSource:0}: Error finding container 2ac7b45aae4dbaf78e6aa4195987afbae9df362ef0b0fbe49dde5a7f700a255c: Status 404 returned error can't find the container with id 2ac7b45aae4dbaf78e6aa4195987afbae9df362ef0b0fbe49dde5a7f700a255c
Sep 09 05:56:59.302702 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:59.302673    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df379a76-12ec-4c26-8dbb-f538d82d9e9f" path="/var/lib/kubelet/pods/df379a76-12ec-4c26-8dbb-f538d82d9e9f/volumes"
Sep 09 05:56:59.569315 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:59.569228    2564 generic.go:356] "Generic (PLEG): container finished" podID="df379a76-12ec-4c26-8dbb-f538d82d9e9f" containerID="8a5c22bf38da029016fa224c1830fe0d6f1a7eaabe9ecee7944959642ae4adc7" exitCode=0
Sep 09 05:56:59.569315 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:59.569272    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lt946"
Sep 09 05:56:59.569574 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:59.569333    2564 scope.go:117] "RemoveContainer" containerID="8a5c22bf38da029016fa224c1830fe0d6f1a7eaabe9ecee7944959642ae4adc7"
Sep 09 05:56:59.571479 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:59.571437    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p" event={"ID":"78dba663-7660-4282-9be7-9647a5f3446f","Type":"ContainerStarted","Data":"b0f49d00f17d2cb751f416ef0e40e1fc23282ac16d7bf4fa660c72961d57d369"}
Sep 09 05:56:59.571633 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:59.571524    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p" event={"ID":"78dba663-7660-4282-9be7-9647a5f3446f","Type":"ContainerStarted","Data":"2ac7b45aae4dbaf78e6aa4195987afbae9df362ef0b0fbe49dde5a7f700a255c"}
Sep 09 05:56:59.571633 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:59.571573    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"
Sep 09 05:56:59.578286 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:59.578271    2564 scope.go:117] "RemoveContainer" containerID="8a5c22bf38da029016fa224c1830fe0d6f1a7eaabe9ecee7944959642ae4adc7"
Sep 09 05:56:59.578567 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:56:59.578550    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5c22bf38da029016fa224c1830fe0d6f1a7eaabe9ecee7944959642ae4adc7\": container with ID starting with 8a5c22bf38da029016fa224c1830fe0d6f1a7eaabe9ecee7944959642ae4adc7 not found: ID does not exist" containerID="8a5c22bf38da029016fa224c1830fe0d6f1a7eaabe9ecee7944959642ae4adc7"
Sep 09 05:56:59.578620 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:59.578574    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5c22bf38da029016fa224c1830fe0d6f1a7eaabe9ecee7944959642ae4adc7"} err="failed to get container status \"8a5c22bf38da029016fa224c1830fe0d6f1a7eaabe9ecee7944959642ae4adc7\": rpc error: code = NotFound desc = could not find container \"8a5c22bf38da029016fa224c1830fe0d6f1a7eaabe9ecee7944959642ae4adc7\": container with ID starting with 8a5c22bf38da029016fa224c1830fe0d6f1a7eaabe9ecee7944959642ae4adc7 not found: ID does not exist"
Sep 09 05:56:59.600726 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:56:59.600677    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p" podStartSLOduration=1.600663046 podStartE2EDuration="1.600663046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:56:59.597929067 +0000 UTC m=+446.814258951" watchObservedRunningTime="2026-09-09 05:56:59.600663046 +0000 UTC m=+446.816992932"
Sep 09 05:57:10.581834 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:10.581796    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"
Sep 09 05:57:14.826744 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:14.826706    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"]
Sep 09 05:57:14.827243 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:14.826980    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p" podUID="78dba663-7660-4282-9be7-9647a5f3446f" containerName="manager" containerID="cri-o://b0f49d00f17d2cb751f416ef0e40e1fc23282ac16d7bf4fa660c72961d57d369" gracePeriod=10
Sep 09 05:57:15.030815 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.030785    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"]
Sep 09 05:57:15.034804 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.034782    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"
Sep 09 05:57:15.050404 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.050377    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"]
Sep 09 05:57:15.077847 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.077794    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"
Sep 09 05:57:15.096862 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.096810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h7fv\" (UniqueName: \"kubernetes.io/projected/353bec02-f547-4b97-a2f6-a0b8d5f4b3fd-kube-api-access-6h7fv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wsnzb\" (UID: \"353bec02-f547-4b97-a2f6-a0b8d5f4b3fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"
Sep 09 05:57:15.097079 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.097026    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/353bec02-f547-4b97-a2f6-a0b8d5f4b3fd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wsnzb\" (UID: \"353bec02-f547-4b97-a2f6-a0b8d5f4b3fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"
Sep 09 05:57:15.198239 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.198199    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xn6v2\" (UniqueName: \"kubernetes.io/projected/78dba663-7660-4282-9be7-9647a5f3446f-kube-api-access-xn6v2\") pod \"78dba663-7660-4282-9be7-9647a5f3446f\" (UID: \"78dba663-7660-4282-9be7-9647a5f3446f\") "
Sep 09 05:57:15.198239 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.198252    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78dba663-7660-4282-9be7-9647a5f3446f-extensions-socket-volume\") pod \"78dba663-7660-4282-9be7-9647a5f3446f\" (UID: \"78dba663-7660-4282-9be7-9647a5f3446f\") "
Sep 09 05:57:15.198477 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.198435    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h7fv\" (UniqueName: \"kubernetes.io/projected/353bec02-f547-4b97-a2f6-a0b8d5f4b3fd-kube-api-access-6h7fv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wsnzb\" (UID: \"353bec02-f547-4b97-a2f6-a0b8d5f4b3fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"
Sep 09 05:57:15.198577 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.198483    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/353bec02-f547-4b97-a2f6-a0b8d5f4b3fd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wsnzb\" (UID: \"353bec02-f547-4b97-a2f6-a0b8d5f4b3fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"
Sep 09 05:57:15.198677 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.198643    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78dba663-7660-4282-9be7-9647a5f3446f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "78dba663-7660-4282-9be7-9647a5f3446f" (UID: "78dba663-7660-4282-9be7-9647a5f3446f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:57:15.198889 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.198872    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/353bec02-f547-4b97-a2f6-a0b8d5f4b3fd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wsnzb\" (UID: \"353bec02-f547-4b97-a2f6-a0b8d5f4b3fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"
Sep 09 05:57:15.200306 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.200283    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78dba663-7660-4282-9be7-9647a5f3446f-kube-api-access-xn6v2" (OuterVolumeSpecName: "kube-api-access-xn6v2") pod "78dba663-7660-4282-9be7-9647a5f3446f" (UID: "78dba663-7660-4282-9be7-9647a5f3446f"). InnerVolumeSpecName "kube-api-access-xn6v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:57:15.206318 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.206297    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h7fv\" (UniqueName: \"kubernetes.io/projected/353bec02-f547-4b97-a2f6-a0b8d5f4b3fd-kube-api-access-6h7fv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wsnzb\" (UID: \"353bec02-f547-4b97-a2f6-a0b8d5f4b3fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"
Sep 09 05:57:15.299051 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.299025    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn6v2\" (UniqueName: \"kubernetes.io/projected/78dba663-7660-4282-9be7-9647a5f3446f-kube-api-access-xn6v2\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:57:15.299051 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.299050    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78dba663-7660-4282-9be7-9647a5f3446f-extensions-socket-volume\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:57:15.345323 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.345245    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"
Sep 09 05:57:15.625731 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.625702    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"]
Sep 09 05:57:15.626651 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:57:15.626624    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353bec02_f547_4b97_a2f6_a0b8d5f4b3fd.slice/crio-1d0117e3f9f8481df17b2ebbaf71ec78ff55ccaff1f4712301b9b43e2ab63b81 WatchSource:0}: Error finding container 1d0117e3f9f8481df17b2ebbaf71ec78ff55ccaff1f4712301b9b43e2ab63b81: Status 404 returned error can't find the container with id 1d0117e3f9f8481df17b2ebbaf71ec78ff55ccaff1f4712301b9b43e2ab63b81
Sep 09 05:57:15.670248 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.670220    2564 generic.go:356] "Generic (PLEG): container finished" podID="78dba663-7660-4282-9be7-9647a5f3446f" containerID="b0f49d00f17d2cb751f416ef0e40e1fc23282ac16d7bf4fa660c72961d57d369" exitCode=0
Sep 09 05:57:15.670393 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.670260    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"
Sep 09 05:57:15.670393 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.670301    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p" event={"ID":"78dba663-7660-4282-9be7-9647a5f3446f","Type":"ContainerDied","Data":"b0f49d00f17d2cb751f416ef0e40e1fc23282ac16d7bf4fa660c72961d57d369"}
Sep 09 05:57:15.670393 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.670360    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p" event={"ID":"78dba663-7660-4282-9be7-9647a5f3446f","Type":"ContainerDied","Data":"2ac7b45aae4dbaf78e6aa4195987afbae9df362ef0b0fbe49dde5a7f700a255c"}
Sep 09 05:57:15.670393 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.670389    2564 scope.go:117] "RemoveContainer" containerID="b0f49d00f17d2cb751f416ef0e40e1fc23282ac16d7bf4fa660c72961d57d369"
Sep 09 05:57:15.672264 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.672238    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb" event={"ID":"353bec02-f547-4b97-a2f6-a0b8d5f4b3fd","Type":"ContainerStarted","Data":"1d0117e3f9f8481df17b2ebbaf71ec78ff55ccaff1f4712301b9b43e2ab63b81"}
Sep 09 05:57:15.679865 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.679842    2564 scope.go:117] "RemoveContainer" containerID="b0f49d00f17d2cb751f416ef0e40e1fc23282ac16d7bf4fa660c72961d57d369"
Sep 09 05:57:15.680141 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:57:15.680124    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f49d00f17d2cb751f416ef0e40e1fc23282ac16d7bf4fa660c72961d57d369\": container with ID starting with b0f49d00f17d2cb751f416ef0e40e1fc23282ac16d7bf4fa660c72961d57d369 not found: ID does not exist" containerID="b0f49d00f17d2cb751f416ef0e40e1fc23282ac16d7bf4fa660c72961d57d369"
Sep 09 05:57:15.680204 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.680149    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f49d00f17d2cb751f416ef0e40e1fc23282ac16d7bf4fa660c72961d57d369"} err="failed to get container status \"b0f49d00f17d2cb751f416ef0e40e1fc23282ac16d7bf4fa660c72961d57d369\": rpc error: code = NotFound desc = could not find container \"b0f49d00f17d2cb751f416ef0e40e1fc23282ac16d7bf4fa660c72961d57d369\": container with ID starting with b0f49d00f17d2cb751f416ef0e40e1fc23282ac16d7bf4fa660c72961d57d369 not found: ID does not exist"
Sep 09 05:57:15.692146 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.692117    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"]
Sep 09 05:57:15.700481 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:15.700458    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d628p"]
Sep 09 05:57:16.680931 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:16.680897    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb" event={"ID":"353bec02-f547-4b97-a2f6-a0b8d5f4b3fd","Type":"ContainerStarted","Data":"a1f41c414ab54a2fa327d7861395c3d793f62c73808d0c4bc759ba18fb34d0a6"}
Sep 09 05:57:16.681296 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:16.680982    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"
Sep 09 05:57:16.704346 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:16.704294    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb" podStartSLOduration=1.70427786 podStartE2EDuration="1.70427786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:57:16.701235245 +0000 UTC m=+463.917565131" watchObservedRunningTime="2026-09-09 05:57:16.70427786 +0000 UTC m=+463.920607743"
Sep 09 05:57:17.302781 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:17.302746    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78dba663-7660-4282-9be7-9647a5f3446f" path="/var/lib/kubelet/pods/78dba663-7660-4282-9be7-9647a5f3446f/volumes"
Sep 09 05:57:27.688627 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:57:27.688598    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"
Sep 09 05:58:22.903222 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:22.903185    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cf64fc486-d68lv"]
Sep 09 05:58:22.903655 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:22.903615    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78dba663-7660-4282-9be7-9647a5f3446f" containerName="manager"
Sep 09 05:58:22.903655 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:22.903628    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="78dba663-7660-4282-9be7-9647a5f3446f" containerName="manager"
Sep 09 05:58:22.903737 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:22.903696    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="78dba663-7660-4282-9be7-9647a5f3446f" containerName="manager"
Sep 09 05:58:22.906761 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:22.906739    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:22.909350 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:22.909324    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 05:58:22.909525 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:22.909403    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 05:58:22.909628 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:22.909412    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 05:58:22.909720 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:22.909457    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-b4428\""
Sep 09 05:58:22.918523 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:22.918482    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cf64fc486-d68lv"]
Sep 09 05:58:23.072705 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.072671    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5945888b8-8rmmg"]
Sep 09 05:58:23.076318 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.076295    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 05:58:23.080532 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.080504    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4xm9\" (UniqueName: \"kubernetes.io/projected/2357febb-8ded-4d33-9aad-bb1215dcaf90-kube-api-access-v4xm9\") pod \"maas-controller-6cf64fc486-d68lv\" (UID: \"2357febb-8ded-4d33-9aad-bb1215dcaf90\") " pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:23.080671 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.080602    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-metrics-certs\") pod \"maas-controller-6cf64fc486-d68lv\" (UID: \"2357febb-8ded-4d33-9aad-bb1215dcaf90\") " pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:23.080671 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.080634    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-webhook-cert\") pod \"maas-controller-6cf64fc486-d68lv\" (UID: \"2357febb-8ded-4d33-9aad-bb1215dcaf90\") " pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:23.087639 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.087615    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5945888b8-8rmmg"]
Sep 09 05:58:23.181594 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.181486    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-webhook-cert\") pod \"maas-controller-6cf64fc486-d68lv\" (UID: \"2357febb-8ded-4d33-9aad-bb1215dcaf90\") " pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:23.181594 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.181566    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1087c96-3a96-44df-bfa9-3e155f2b287a-metrics-certs\") pod \"maas-controller-5945888b8-8rmmg\" (UID: \"c1087c96-3a96-44df-bfa9-3e155f2b287a\") " pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 05:58:23.181594 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.181590    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4xm9\" (UniqueName: \"kubernetes.io/projected/2357febb-8ded-4d33-9aad-bb1215dcaf90-kube-api-access-v4xm9\") pod \"maas-controller-6cf64fc486-d68lv\" (UID: \"2357febb-8ded-4d33-9aad-bb1215dcaf90\") " pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:23.181853 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.181629    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jps\" (UniqueName: \"kubernetes.io/projected/c1087c96-3a96-44df-bfa9-3e155f2b287a-kube-api-access-97jps\") pod \"maas-controller-5945888b8-8rmmg\" (UID: \"c1087c96-3a96-44df-bfa9-3e155f2b287a\") " pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 05:58:23.181853 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:58:23.181636    2564 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 09 05:58:23.181853 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:58:23.181699    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-webhook-cert podName:2357febb-8ded-4d33-9aad-bb1215dcaf90 nodeName:}" failed. No retries permitted until 2026-09-09 05:58:23.681683707 +0000 UTC m=+530.898013568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-webhook-cert") pod "maas-controller-6cf64fc486-d68lv" (UID: "2357febb-8ded-4d33-9aad-bb1215dcaf90") : secret "maas-controller-webhook-cert" not found
Sep 09 05:58:23.181853 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.181721    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1087c96-3a96-44df-bfa9-3e155f2b287a-webhook-cert\") pod \"maas-controller-5945888b8-8rmmg\" (UID: \"c1087c96-3a96-44df-bfa9-3e155f2b287a\") " pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 05:58:23.181853 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.181762    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-metrics-certs\") pod \"maas-controller-6cf64fc486-d68lv\" (UID: \"2357febb-8ded-4d33-9aad-bb1215dcaf90\") " pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:23.181853 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:58:23.181844    2564 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 09 05:58:23.182046 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:58:23.181884    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-metrics-certs podName:2357febb-8ded-4d33-9aad-bb1215dcaf90 nodeName:}" failed. No retries permitted until 2026-09-09 05:58:23.681872417 +0000 UTC m=+530.898202279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-metrics-certs") pod "maas-controller-6cf64fc486-d68lv" (UID: "2357febb-8ded-4d33-9aad-bb1215dcaf90") : secret "maas-controller-metrics-tls" not found
Sep 09 05:58:23.189617 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.189594    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4xm9\" (UniqueName: \"kubernetes.io/projected/2357febb-8ded-4d33-9aad-bb1215dcaf90-kube-api-access-v4xm9\") pod \"maas-controller-6cf64fc486-d68lv\" (UID: \"2357febb-8ded-4d33-9aad-bb1215dcaf90\") " pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:23.282401 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.282367    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jps\" (UniqueName: \"kubernetes.io/projected/c1087c96-3a96-44df-bfa9-3e155f2b287a-kube-api-access-97jps\") pod \"maas-controller-5945888b8-8rmmg\" (UID: \"c1087c96-3a96-44df-bfa9-3e155f2b287a\") " pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 05:58:23.282604 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.282428    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1087c96-3a96-44df-bfa9-3e155f2b287a-webhook-cert\") pod \"maas-controller-5945888b8-8rmmg\" (UID: \"c1087c96-3a96-44df-bfa9-3e155f2b287a\") " pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 05:58:23.282604 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.282524    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1087c96-3a96-44df-bfa9-3e155f2b287a-metrics-certs\") pod \"maas-controller-5945888b8-8rmmg\" (UID: \"c1087c96-3a96-44df-bfa9-3e155f2b287a\") " pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 05:58:23.282687 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:58:23.282629    2564 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 09 05:58:23.282722 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:58:23.282689    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1087c96-3a96-44df-bfa9-3e155f2b287a-metrics-certs podName:c1087c96-3a96-44df-bfa9-3e155f2b287a nodeName:}" failed. No retries permitted until 2026-09-09 05:58:23.782674506 +0000 UTC m=+530.999004369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1087c96-3a96-44df-bfa9-3e155f2b287a-metrics-certs") pod "maas-controller-5945888b8-8rmmg" (UID: "c1087c96-3a96-44df-bfa9-3e155f2b287a") : secret "maas-controller-metrics-tls" not found
Sep 09 05:58:23.285032 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.285006    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1087c96-3a96-44df-bfa9-3e155f2b287a-webhook-cert\") pod \"maas-controller-5945888b8-8rmmg\" (UID: \"c1087c96-3a96-44df-bfa9-3e155f2b287a\") " pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 05:58:23.290950 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.290929    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jps\" (UniqueName: \"kubernetes.io/projected/c1087c96-3a96-44df-bfa9-3e155f2b287a-kube-api-access-97jps\") pod \"maas-controller-5945888b8-8rmmg\" (UID: \"c1087c96-3a96-44df-bfa9-3e155f2b287a\") " pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 05:58:23.686474 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.686441    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-metrics-certs\") pod \"maas-controller-6cf64fc486-d68lv\" (UID: \"2357febb-8ded-4d33-9aad-bb1215dcaf90\") " pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:23.686474 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.686480    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-webhook-cert\") pod \"maas-controller-6cf64fc486-d68lv\" (UID: \"2357febb-8ded-4d33-9aad-bb1215dcaf90\") " pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:23.688977 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.688947    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-metrics-certs\") pod \"maas-controller-6cf64fc486-d68lv\" (UID: \"2357febb-8ded-4d33-9aad-bb1215dcaf90\") " pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:23.689166 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.688951    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-webhook-cert\") pod \"maas-controller-6cf64fc486-d68lv\" (UID: \"2357febb-8ded-4d33-9aad-bb1215dcaf90\") " pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:23.787404 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.787363    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1087c96-3a96-44df-bfa9-3e155f2b287a-metrics-certs\") pod \"maas-controller-5945888b8-8rmmg\" (UID: \"c1087c96-3a96-44df-bfa9-3e155f2b287a\") " pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 05:58:23.789730 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.789710    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1087c96-3a96-44df-bfa9-3e155f2b287a-metrics-certs\") pod \"maas-controller-5945888b8-8rmmg\" (UID: \"c1087c96-3a96-44df-bfa9-3e155f2b287a\") " pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 05:58:23.818770 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.818739    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:23.988567 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:23.988452    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 05:58:24.090237 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:24.090209    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cf64fc486-d68lv"]
Sep 09 05:58:24.090905 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:58:24.090881    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2357febb_8ded_4d33_9aad_bb1215dcaf90.slice/crio-925d3b297feace7d0085ed123aaa9b0da1a95dbe23e827cb527e42de18734e88 WatchSource:0}: Error finding container 925d3b297feace7d0085ed123aaa9b0da1a95dbe23e827cb527e42de18734e88: Status 404 returned error can't find the container with id 925d3b297feace7d0085ed123aaa9b0da1a95dbe23e827cb527e42de18734e88
Sep 09 05:58:24.260101 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:24.260066    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5945888b8-8rmmg"]
Sep 09 05:58:24.261383 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:58:24.261354    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1087c96_3a96_44df_bfa9_3e155f2b287a.slice/crio-d172e8d6e5b1ededd4543743cc870b9294417c8fb8ec4bd0458e230f6aaa0990 WatchSource:0}: Error finding container d172e8d6e5b1ededd4543743cc870b9294417c8fb8ec4bd0458e230f6aaa0990: Status 404 returned error can't find the container with id d172e8d6e5b1ededd4543743cc870b9294417c8fb8ec4bd0458e230f6aaa0990
Sep 09 05:58:25.079055 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:25.079013    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5945888b8-8rmmg" event={"ID":"c1087c96-3a96-44df-bfa9-3e155f2b287a","Type":"ContainerStarted","Data":"d172e8d6e5b1ededd4543743cc870b9294417c8fb8ec4bd0458e230f6aaa0990"}
Sep 09 05:58:25.082692 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:25.082654    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cf64fc486-d68lv" event={"ID":"2357febb-8ded-4d33-9aad-bb1215dcaf90","Type":"ContainerStarted","Data":"925d3b297feace7d0085ed123aaa9b0da1a95dbe23e827cb527e42de18734e88"}
Sep 09 05:58:27.096781 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:27.096738    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cf64fc486-d68lv" event={"ID":"2357febb-8ded-4d33-9aad-bb1215dcaf90","Type":"ContainerStarted","Data":"927443d33095b95411dc5f4915331743126e3b2681218b65905c97bf3708e7c1"}
Sep 09 05:58:27.097237 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:27.096813    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:27.098307 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:27.098284    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5945888b8-8rmmg" event={"ID":"c1087c96-3a96-44df-bfa9-3e155f2b287a","Type":"ContainerStarted","Data":"9381b68ec6c093a272930530e36b6ff8497bf152c03f01f94ef62744f7284e25"}
Sep 09 05:58:27.098436 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:27.098409    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 05:58:27.121551 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:27.121508    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cf64fc486-d68lv" podStartSLOduration=2.416783254 podStartE2EDuration="5.121478844s" totalImagesPullingTime="2.70469559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:58:27.118916327 +0000 UTC m=+534.335246214" watchObservedRunningTime="2026-09-09 05:58:27.121478844 +0000 UTC m=+534.337808774"
Sep 09 05:58:27.142159 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:27.142109    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5945888b8-8rmmg" podStartSLOduration=1.601517097 podStartE2EDuration="4.142094245s" totalImagesPullingTime="2.540577148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:58:27.141092852 +0000 UTC m=+534.357422750" watchObservedRunningTime="2026-09-09 05:58:27.142094245 +0000 UTC m=+534.358424128"
Sep 09 05:58:38.106416 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:38.106384    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 05:58:38.106859 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:38.106455    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:38.172554 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:38.172522    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cf64fc486-d68lv"]
Sep 09 05:58:38.172773 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:38.172743    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6cf64fc486-d68lv" podUID="2357febb-8ded-4d33-9aad-bb1215dcaf90" containerName="manager" containerID="cri-o://927443d33095b95411dc5f4915331743126e3b2681218b65905c97bf3708e7c1" gracePeriod=30
Sep 09 05:58:38.424753 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:38.424729    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:38.504413 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:38.504372    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4xm9\" (UniqueName: \"kubernetes.io/projected/2357febb-8ded-4d33-9aad-bb1215dcaf90-kube-api-access-v4xm9\") pod \"2357febb-8ded-4d33-9aad-bb1215dcaf90\" (UID: \"2357febb-8ded-4d33-9aad-bb1215dcaf90\") "
Sep 09 05:58:38.504621 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:38.504475    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-webhook-cert\") pod \"2357febb-8ded-4d33-9aad-bb1215dcaf90\" (UID: \"2357febb-8ded-4d33-9aad-bb1215dcaf90\") "
Sep 09 05:58:38.504621 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:38.504515    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-metrics-certs\") pod \"2357febb-8ded-4d33-9aad-bb1215dcaf90\" (UID: \"2357febb-8ded-4d33-9aad-bb1215dcaf90\") "
Sep 09 05:58:38.506652 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:38.506622    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2357febb-8ded-4d33-9aad-bb1215dcaf90-kube-api-access-v4xm9" (OuterVolumeSpecName: "kube-api-access-v4xm9") pod "2357febb-8ded-4d33-9aad-bb1215dcaf90" (UID: "2357febb-8ded-4d33-9aad-bb1215dcaf90"). InnerVolumeSpecName "kube-api-access-v4xm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:58:38.506652 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:38.506637    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2357febb-8ded-4d33-9aad-bb1215dcaf90" (UID: "2357febb-8ded-4d33-9aad-bb1215dcaf90"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:58:38.506800 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:38.506658    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2357febb-8ded-4d33-9aad-bb1215dcaf90" (UID: "2357febb-8ded-4d33-9aad-bb1215dcaf90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:58:38.605780 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:38.605735    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4xm9\" (UniqueName: \"kubernetes.io/projected/2357febb-8ded-4d33-9aad-bb1215dcaf90-kube-api-access-v4xm9\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:58:38.605780 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:38.605774    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-webhook-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:58:38.605780 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:38.605783    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2357febb-8ded-4d33-9aad-bb1215dcaf90-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 05:58:39.169229 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:39.169192    2564 generic.go:356] "Generic (PLEG): container finished" podID="2357febb-8ded-4d33-9aad-bb1215dcaf90" containerID="927443d33095b95411dc5f4915331743126e3b2681218b65905c97bf3708e7c1" exitCode=0
Sep 09 05:58:39.169668 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:39.169241    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cf64fc486-d68lv"
Sep 09 05:58:39.169668 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:39.169274    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cf64fc486-d68lv" event={"ID":"2357febb-8ded-4d33-9aad-bb1215dcaf90","Type":"ContainerDied","Data":"927443d33095b95411dc5f4915331743126e3b2681218b65905c97bf3708e7c1"}
Sep 09 05:58:39.169668 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:39.169332    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cf64fc486-d68lv" event={"ID":"2357febb-8ded-4d33-9aad-bb1215dcaf90","Type":"ContainerDied","Data":"925d3b297feace7d0085ed123aaa9b0da1a95dbe23e827cb527e42de18734e88"}
Sep 09 05:58:39.169668 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:39.169355    2564 scope.go:117] "RemoveContainer" containerID="927443d33095b95411dc5f4915331743126e3b2681218b65905c97bf3708e7c1"
Sep 09 05:58:39.179011 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:39.178852    2564 scope.go:117] "RemoveContainer" containerID="927443d33095b95411dc5f4915331743126e3b2681218b65905c97bf3708e7c1"
Sep 09 05:58:39.179139 ip-10-0-131-250 kubenswrapper[2564]: E0909 05:58:39.179121    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927443d33095b95411dc5f4915331743126e3b2681218b65905c97bf3708e7c1\": container with ID starting with 927443d33095b95411dc5f4915331743126e3b2681218b65905c97bf3708e7c1 not found: ID does not exist" containerID="927443d33095b95411dc5f4915331743126e3b2681218b65905c97bf3708e7c1"
Sep 09 05:58:39.179188 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:39.179148    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927443d33095b95411dc5f4915331743126e3b2681218b65905c97bf3708e7c1"} err="failed to get container status \"927443d33095b95411dc5f4915331743126e3b2681218b65905c97bf3708e7c1\": rpc error: code = NotFound desc = could not find container \"927443d33095b95411dc5f4915331743126e3b2681218b65905c97bf3708e7c1\": container with ID starting with 927443d33095b95411dc5f4915331743126e3b2681218b65905c97bf3708e7c1 not found: ID does not exist"
Sep 09 05:58:39.190393 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:39.190370    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cf64fc486-d68lv"]
Sep 09 05:58:39.196943 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:39.196921    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6cf64fc486-d68lv"]
Sep 09 05:58:39.302941 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:58:39.302912    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2357febb-8ded-4d33-9aad-bb1215dcaf90" path="/var/lib/kubelet/pods/2357febb-8ded-4d33-9aad-bb1215dcaf90/volumes"
Sep 09 05:59:00.404981 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.404950    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp"]
Sep 09 05:59:00.405432 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.405345    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2357febb-8ded-4d33-9aad-bb1215dcaf90" containerName="manager"
Sep 09 05:59:00.405432 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.405356    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2357febb-8ded-4d33-9aad-bb1215dcaf90" containerName="manager"
Sep 09 05:59:00.405432 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.405421    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2357febb-8ded-4d33-9aad-bb1215dcaf90" containerName="manager"
Sep 09 05:59:00.408436 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.408419    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp"
Sep 09 05:59:00.411048 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.411017    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 05:59:00.411184 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.411115    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4pp9f\""
Sep 09 05:59:00.425131 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.425101    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp"]
Sep 09 05:59:00.504742 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.504706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dddcc55b-5388-47fa-bdae-00c80c98d4a4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6fltp\" (UID: \"dddcc55b-5388-47fa-bdae-00c80c98d4a4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp"
Sep 09 05:59:00.504910 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.504756    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqzm\" (UniqueName: \"kubernetes.io/projected/dddcc55b-5388-47fa-bdae-00c80c98d4a4-kube-api-access-6zqzm\") pod \"payload-pre-processing-59fbc47bf8-6fltp\" (UID: \"dddcc55b-5388-47fa-bdae-00c80c98d4a4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp"
Sep 09 05:59:00.605923 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.605883    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dddcc55b-5388-47fa-bdae-00c80c98d4a4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6fltp\" (UID: \"dddcc55b-5388-47fa-bdae-00c80c98d4a4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp"
Sep 09 05:59:00.606126 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.605939    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqzm\" (UniqueName: \"kubernetes.io/projected/dddcc55b-5388-47fa-bdae-00c80c98d4a4-kube-api-access-6zqzm\") pod \"payload-pre-processing-59fbc47bf8-6fltp\" (UID: \"dddcc55b-5388-47fa-bdae-00c80c98d4a4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp"
Sep 09 05:59:00.606685 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.606658    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dddcc55b-5388-47fa-bdae-00c80c98d4a4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6fltp\" (UID: \"dddcc55b-5388-47fa-bdae-00c80c98d4a4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp"
Sep 09 05:59:00.620138 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.620108    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqzm\" (UniqueName: \"kubernetes.io/projected/dddcc55b-5388-47fa-bdae-00c80c98d4a4-kube-api-access-6zqzm\") pod \"payload-pre-processing-59fbc47bf8-6fltp\" (UID: \"dddcc55b-5388-47fa-bdae-00c80c98d4a4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp"
Sep 09 05:59:00.724431 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.724352    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp"
Sep 09 05:59:00.999028 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:00.999000    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp"]
Sep 09 05:59:01.000809 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:59:01.000781    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddddcc55b_5388_47fa_bdae_00c80c98d4a4.slice/crio-da004a9cbdceccfa6570b646a413ac725b6c82ad4d7a58b676de048f80e49b8e WatchSource:0}: Error finding container da004a9cbdceccfa6570b646a413ac725b6c82ad4d7a58b676de048f80e49b8e: Status 404 returned error can't find the container with id da004a9cbdceccfa6570b646a413ac725b6c82ad4d7a58b676de048f80e49b8e
Sep 09 05:59:01.295712 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:01.295624    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp" event={"ID":"dddcc55b-5388-47fa-bdae-00c80c98d4a4","Type":"ContainerStarted","Data":"da004a9cbdceccfa6570b646a413ac725b6c82ad4d7a58b676de048f80e49b8e"}
Sep 09 05:59:03.310452 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:03.310414    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp" event={"ID":"dddcc55b-5388-47fa-bdae-00c80c98d4a4","Type":"ContainerStarted","Data":"b30371835c40a66750e92addb29098ea112c0d7b177e1c8871ec5bbeed4c240d"}
Sep 09 05:59:03.310909 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:03.310477    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp"
Sep 09 05:59:03.326797 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:03.326752    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp" podStartSLOduration=1.6441677289999999 podStartE2EDuration="3.326736168s" totalImagesPullingTime="1.682568439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:59:03.324735963 +0000 UTC m=+570.541065848" watchObservedRunningTime="2026-09-09 05:59:03.326736168 +0000 UTC m=+570.543066085"
Sep 09 05:59:14.318132 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:14.318094    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6fltp"
Sep 09 05:59:21.335379 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.335296    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"]
Sep 09 05:59:21.341925 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.341896    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.345099 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.345060    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 05:59:21.345099 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.345085    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 05:59:21.345327 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.345147    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 09 05:59:21.345394 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.345358    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w84m4\""
Sep 09 05:59:21.347897 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.347482    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"]
Sep 09 05:59:21.487825 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.487791    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.488012 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.487950    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.488012 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.487990    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2flmj\" (UniqueName: \"kubernetes.io/projected/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-kube-api-access-2flmj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.488146 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.488079    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.488205 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.488162    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.488205 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.488196    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.588978 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.588900    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.588978 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.588937    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.589194 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.589066    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.589194 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.589124    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.589194 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.589161    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2flmj\" (UniqueName: \"kubernetes.io/projected/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-kube-api-access-2flmj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.589347 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.589207    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.589347 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.589310    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.589453 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.589354    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.589612 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.589592    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.591303 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.591280    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.591474 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.591458    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.596970 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.596947    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2flmj\" (UniqueName: \"kubernetes.io/projected/b29a44b5-1e98-4569-abf1-aaa167a7ea2e-kube-api-access-2flmj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7\" (UID: \"b29a44b5-1e98-4569-abf1-aaa167a7ea2e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.654945 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.654911    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:21.933340 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:21.933315    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"]
Sep 09 05:59:21.935210 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:59:21.935180    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29a44b5_1e98_4569_abf1_aaa167a7ea2e.slice/crio-bba11a742c6ad95c9a1009eb6acc75cf868bc631402d7c9d2f1844a0c10483b0 WatchSource:0}: Error finding container bba11a742c6ad95c9a1009eb6acc75cf868bc631402d7c9d2f1844a0c10483b0: Status 404 returned error can't find the container with id bba11a742c6ad95c9a1009eb6acc75cf868bc631402d7c9d2f1844a0c10483b0
Sep 09 05:59:22.419350 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:22.419306    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7" event={"ID":"b29a44b5-1e98-4569-abf1-aaa167a7ea2e","Type":"ContainerStarted","Data":"bba11a742c6ad95c9a1009eb6acc75cf868bc631402d7c9d2f1844a0c10483b0"}
Sep 09 05:59:23.426005 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:23.425968    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7" event={"ID":"b29a44b5-1e98-4569-abf1-aaa167a7ea2e","Type":"ContainerStarted","Data":"a54292ba3a4d5ece464a4bdfc487e0792be001aec8d0051fc5d97d1fb593b4aa"}
Sep 09 05:59:23.446172 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:23.446120    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7" podStartSLOduration=1.36488381 podStartE2EDuration="2.446104086s" totalImagesPullingTime="1.081220276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:59:23.44393287 +0000 UTC m=+590.660262756" watchObservedRunningTime="2026-09-09 05:59:23.446104086 +0000 UTC m=+590.662433971"
Sep 09 05:59:31.655255 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:31.655215    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:31.655686 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:31.655291    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:31.667837 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:31.667812    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:32.489821 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:32.489784    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7"
Sep 09 05:59:32.919931 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:32.919901    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"]
Sep 09 05:59:32.923597 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:32.923571    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:32.926038 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:32.926014    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 09 05:59:32.933020 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:32.932994    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"]
Sep 09 05:59:32.984661 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:32.984624    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3044f77-f03a-4d81-8881-efce640da8a8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:32.984661 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:32.984661    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3044f77-f03a-4d81-8881-efce640da8a8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:32.984864 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:32.984691    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3044f77-f03a-4d81-8881-efce640da8a8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:32.984864 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:32.984723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3044f77-f03a-4d81-8881-efce640da8a8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:32.984864 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:32.984749    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3044f77-f03a-4d81-8881-efce640da8a8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:32.984864 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:32.984852    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz5cq\" (UniqueName: \"kubernetes.io/projected/e3044f77-f03a-4d81-8881-efce640da8a8-kube-api-access-lz5cq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:33.085967 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.085927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3044f77-f03a-4d81-8881-efce640da8a8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:33.085967 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.085974    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3044f77-f03a-4d81-8881-efce640da8a8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:33.086220 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.086032    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz5cq\" (UniqueName: \"kubernetes.io/projected/e3044f77-f03a-4d81-8881-efce640da8a8-kube-api-access-lz5cq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:33.086220 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.086076    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3044f77-f03a-4d81-8881-efce640da8a8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:33.086220 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.086099    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3044f77-f03a-4d81-8881-efce640da8a8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:33.086220 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.086127    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3044f77-f03a-4d81-8881-efce640da8a8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:33.086531 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.086477    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3044f77-f03a-4d81-8881-efce640da8a8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:33.086614 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.086533    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3044f77-f03a-4d81-8881-efce640da8a8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:33.086614 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.086589    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3044f77-f03a-4d81-8881-efce640da8a8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:33.088312 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.088289    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3044f77-f03a-4d81-8881-efce640da8a8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:33.088595 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.088578    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3044f77-f03a-4d81-8881-efce640da8a8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:33.094237 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.094213    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz5cq\" (UniqueName: \"kubernetes.io/projected/e3044f77-f03a-4d81-8881-efce640da8a8-kube-api-access-lz5cq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bbqs4\" (UID: \"e3044f77-f03a-4d81-8881-efce640da8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:33.236139 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.236064    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:33.263517 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.263449    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 05:59:33.264035 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.264011    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 05:59:33.520055 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:33.520028    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"]
Sep 09 05:59:33.521505 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:59:33.521462    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3044f77_f03a_4d81_8881_efce640da8a8.slice/crio-1e09bbcf5e9a0abbd1486a5cdf02f4b759224c555235bf4c767c96389bba220b WatchSource:0}: Error finding container 1e09bbcf5e9a0abbd1486a5cdf02f4b759224c555235bf4c767c96389bba220b: Status 404 returned error can't find the container with id 1e09bbcf5e9a0abbd1486a5cdf02f4b759224c555235bf4c767c96389bba220b
Sep 09 05:59:34.490921 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:34.490883    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4" event={"ID":"e3044f77-f03a-4d81-8881-efce640da8a8","Type":"ContainerStarted","Data":"93227d80e4fd02c59d92de01d8302fdd9304d9a8cb213899e345c0932c93470f"}
Sep 09 05:59:34.491324 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:34.490965    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4" event={"ID":"e3044f77-f03a-4d81-8881-efce640da8a8","Type":"ContainerStarted","Data":"1e09bbcf5e9a0abbd1486a5cdf02f4b759224c555235bf4c767c96389bba220b"}
Sep 09 05:59:34.516064 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:34.516015    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4" podStartSLOduration=2.336335557 podStartE2EDuration="2.515999197s" totalImagesPullingTime="179.66364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:59:34.514101357 +0000 UTC m=+601.730431242" watchObservedRunningTime="2026-09-09 05:59:34.515999197 +0000 UTC m=+601.732329081"
Sep 09 05:59:43.236514 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:43.236461    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:43.238999 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:43.236530    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:43.249144 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:43.249119    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:43.553949 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:43.553868    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bbqs4"
Sep 09 05:59:52.936835 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:52.936795    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"]
Sep 09 05:59:52.941217 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:52.941190    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:52.944628 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:52.944603    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 09 05:59:52.947517 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:52.947469    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"]
Sep 09 05:59:53.074009 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.073975    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b9245f-4729-42eb-8ed4-112a6f5a3490-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.074009 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.074010    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14b9245f-4729-42eb-8ed4-112a6f5a3490-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.074212 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.074160    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b9245f-4729-42eb-8ed4-112a6f5a3490-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.074212 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.074198    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14b9245f-4729-42eb-8ed4-112a6f5a3490-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.074284 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.074226    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b9245f-4729-42eb-8ed4-112a6f5a3490-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.074284 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.074260    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8nl5\" (UniqueName: \"kubernetes.io/projected/14b9245f-4729-42eb-8ed4-112a6f5a3490-kube-api-access-z8nl5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.175784 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.175742    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b9245f-4729-42eb-8ed4-112a6f5a3490-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.175784 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.175788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14b9245f-4729-42eb-8ed4-112a6f5a3490-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.176001 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.175904    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b9245f-4729-42eb-8ed4-112a6f5a3490-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.176001 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.175944    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8nl5\" (UniqueName: \"kubernetes.io/projected/14b9245f-4729-42eb-8ed4-112a6f5a3490-kube-api-access-z8nl5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.176001 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.175988    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b9245f-4729-42eb-8ed4-112a6f5a3490-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.176149 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.176015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14b9245f-4729-42eb-8ed4-112a6f5a3490-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.176204 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.176156    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14b9245f-4729-42eb-8ed4-112a6f5a3490-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.176289 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.176262    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b9245f-4729-42eb-8ed4-112a6f5a3490-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.176354 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.176301    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b9245f-4729-42eb-8ed4-112a6f5a3490-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.177949 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.177928    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b9245f-4729-42eb-8ed4-112a6f5a3490-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.178253 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.178235    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14b9245f-4729-42eb-8ed4-112a6f5a3490-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.185409 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.185384    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8nl5\" (UniqueName: \"kubernetes.io/projected/14b9245f-4729-42eb-8ed4-112a6f5a3490-kube-api-access-z8nl5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrmmf\" (UID: \"14b9245f-4729-42eb-8ed4-112a6f5a3490\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.254132 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.254042    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 05:59:53.531238 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.531206    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"]
Sep 09 05:59:53.532699 ip-10-0-131-250 kubenswrapper[2564]: W0909 05:59:53.532671    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b9245f_4729_42eb_8ed4_112a6f5a3490.slice/crio-5d438d0958f9e4075feedb54af82002f07a573ebba8aec17a5893c3156bad947 WatchSource:0}: Error finding container 5d438d0958f9e4075feedb54af82002f07a573ebba8aec17a5893c3156bad947: Status 404 returned error can't find the container with id 5d438d0958f9e4075feedb54af82002f07a573ebba8aec17a5893c3156bad947
Sep 09 05:59:53.534105 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.534086    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 05:59:53.608075 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:53.608039    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf" event={"ID":"14b9245f-4729-42eb-8ed4-112a6f5a3490","Type":"ContainerStarted","Data":"5d438d0958f9e4075feedb54af82002f07a573ebba8aec17a5893c3156bad947"}
Sep 09 05:59:54.614806 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:54.614769    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf" event={"ID":"14b9245f-4729-42eb-8ed4-112a6f5a3490","Type":"ContainerStarted","Data":"6914f7a1398248458b0c8c9d3f2aae97be789e33937a70fcaf1a17678a584d30"}
Sep 09 05:59:54.633748 ip-10-0-131-250 kubenswrapper[2564]: I0909 05:59:54.633696    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf" podStartSLOduration=2.4622031 podStartE2EDuration="2.633679538s" totalImagesPullingTime="171.476438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:59:54.631120138 +0000 UTC m=+621.847450035" watchObservedRunningTime="2026-09-09 05:59:54.633679538 +0000 UTC m=+621.850009426"
Sep 09 06:00:03.255227 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:00:03.255140    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 06:00:03.255227 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:00:03.255208    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 06:00:03.268236 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:00:03.268199    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 06:00:03.678702 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:00:03.678672    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrmmf"
Sep 09 06:01:52.833751 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:52.833717    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"]
Sep 09 06:01:52.837503 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:52.837477    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"
Sep 09 06:01:52.839959 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:52.839936    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-43eb2d\""
Sep 09 06:01:52.840107 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:52.840056    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-43eb2d-dockercfg-ljbvt\""
Sep 09 06:01:52.849370 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:52.849346    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"]
Sep 09 06:01:52.878088 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:52.878061    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"]
Sep 09 06:01:52.881664 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:52.881644    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"
Sep 09 06:01:52.892357 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:52.892333    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"]
Sep 09 06:01:52.965680 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:52.965646    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe57a46-2294-467f-93a2-d1386a262c69-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq\" (UID: \"bfe57a46-2294-467f-93a2-d1386a262c69\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"
Sep 09 06:01:52.965845 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:52.965715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvs6\" (UniqueName: \"kubernetes.io/projected/bfe57a46-2294-467f-93a2-d1386a262c69-kube-api-access-mmvs6\") pod \"payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq\" (UID: \"bfe57a46-2294-467f-93a2-d1386a262c69\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"
Sep 09 06:01:52.965845 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:52.965764    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvlg\" (UniqueName: \"kubernetes.io/projected/94d23818-c30b-4b0a-8469-3575669b2bf9-kube-api-access-vfvlg\") pod \"payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf\" (UID: \"94d23818-c30b-4b0a-8469-3575669b2bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"
Sep 09 06:01:52.965929 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:52.965875    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d23818-c30b-4b0a-8469-3575669b2bf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf\" (UID: \"94d23818-c30b-4b0a-8469-3575669b2bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"
Sep 09 06:01:53.066429 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.066393    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe57a46-2294-467f-93a2-d1386a262c69-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq\" (UID: \"bfe57a46-2294-467f-93a2-d1386a262c69\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"
Sep 09 06:01:53.066429 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.066436    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvs6\" (UniqueName: \"kubernetes.io/projected/bfe57a46-2294-467f-93a2-d1386a262c69-kube-api-access-mmvs6\") pod \"payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq\" (UID: \"bfe57a46-2294-467f-93a2-d1386a262c69\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"
Sep 09 06:01:53.066709 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.066455    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvlg\" (UniqueName: \"kubernetes.io/projected/94d23818-c30b-4b0a-8469-3575669b2bf9-kube-api-access-vfvlg\") pod \"payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf\" (UID: \"94d23818-c30b-4b0a-8469-3575669b2bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"
Sep 09 06:01:53.066709 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.066534    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d23818-c30b-4b0a-8469-3575669b2bf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf\" (UID: \"94d23818-c30b-4b0a-8469-3575669b2bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"
Sep 09 06:01:53.067260 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.067241    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d23818-c30b-4b0a-8469-3575669b2bf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf\" (UID: \"94d23818-c30b-4b0a-8469-3575669b2bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"
Sep 09 06:01:53.067443 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.067417    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe57a46-2294-467f-93a2-d1386a262c69-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq\" (UID: \"bfe57a46-2294-467f-93a2-d1386a262c69\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"
Sep 09 06:01:53.076219 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.076197    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvs6\" (UniqueName: \"kubernetes.io/projected/bfe57a46-2294-467f-93a2-d1386a262c69-kube-api-access-mmvs6\") pod \"payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq\" (UID: \"bfe57a46-2294-467f-93a2-d1386a262c69\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"
Sep 09 06:01:53.076332 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.076294    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvlg\" (UniqueName: \"kubernetes.io/projected/94d23818-c30b-4b0a-8469-3575669b2bf9-kube-api-access-vfvlg\") pod \"payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf\" (UID: \"94d23818-c30b-4b0a-8469-3575669b2bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"
Sep 09 06:01:53.149393 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.149308    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"
Sep 09 06:01:53.193433 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.193165    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"
Sep 09 06:01:53.222154 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.222120    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"]
Sep 09 06:01:53.226723 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.226700    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:53.230081 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.230059    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 06:01:53.230632 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.230612    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ecd13e-serving-cert\""
Sep 09 06:01:53.231203 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.231147    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"]
Sep 09 06:01:53.231659 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.231543    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 06:01:53.235539 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.234808    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 06:01:53.235665 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.235588    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-927tl\""
Sep 09 06:01:53.269805 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.269031    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6263b891-ad1a-4c05-9754-5c75cb38b50c-metrics-certs\") pod \"maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:53.269805 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.269087    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6263b891-ad1a-4c05-9754-5c75cb38b50c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:53.269805 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.269121    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll4dj\" (UniqueName: \"kubernetes.io/projected/6263b891-ad1a-4c05-9754-5c75cb38b50c-kube-api-access-ll4dj\") pod \"maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:53.269805 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.269173    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6263b891-ad1a-4c05-9754-5c75cb38b50c-maas-api-tls\") pod \"maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:53.370104 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.370074    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6263b891-ad1a-4c05-9754-5c75cb38b50c-metrics-certs\") pod \"maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:53.370294 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.370122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6263b891-ad1a-4c05-9754-5c75cb38b50c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:53.370294 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.370146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll4dj\" (UniqueName: \"kubernetes.io/projected/6263b891-ad1a-4c05-9754-5c75cb38b50c-kube-api-access-ll4dj\") pod \"maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:53.370294 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.370184    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6263b891-ad1a-4c05-9754-5c75cb38b50c-maas-api-tls\") pod \"maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:53.370481 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:01:53.370323    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-serving-cert: secret "maas-api-e2e-inf-a-ecd13e-serving-cert" not found
Sep 09 06:01:53.370481 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:01:53.370387    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6263b891-ad1a-4c05-9754-5c75cb38b50c-maas-api-tls podName:6263b891-ad1a-4c05-9754-5c75cb38b50c nodeName:}" failed. No retries permitted until 2026-09-09 06:01:53.870371629 +0000 UTC m=+741.086701491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6263b891-ad1a-4c05-9754-5c75cb38b50c-maas-api-tls") pod "maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5" (UID: "6263b891-ad1a-4c05-9754-5c75cb38b50c") : secret "maas-api-e2e-inf-a-ecd13e-serving-cert" not found
Sep 09 06:01:53.370863 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.370828    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6263b891-ad1a-4c05-9754-5c75cb38b50c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:53.372646 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.372626    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6263b891-ad1a-4c05-9754-5c75cb38b50c-metrics-certs\") pod \"maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:53.378462 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.378444    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll4dj\" (UniqueName: \"kubernetes.io/projected/6263b891-ad1a-4c05-9754-5c75cb38b50c-kube-api-access-ll4dj\") pod \"maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:53.435456 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.435432    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"]
Sep 09 06:01:53.436879 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:01:53.436848    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe57a46_2294_467f_93a2_d1386a262c69.slice/crio-528363aedbaa6b0b925514c8280c6df331a834e6375d9657bb364578ce06d68f WatchSource:0}: Error finding container 528363aedbaa6b0b925514c8280c6df331a834e6375d9657bb364578ce06d68f: Status 404 returned error can't find the container with id 528363aedbaa6b0b925514c8280c6df331a834e6375d9657bb364578ce06d68f
Sep 09 06:01:53.492648 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.492625    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"]
Sep 09 06:01:53.499782 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:01:53.499756    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d23818_c30b_4b0a_8469_3575669b2bf9.slice/crio-088d61a5706970cfa328fb9eeff8bb8b9d9db1c33edaa6dcdf46d75786fd0ae3 WatchSource:0}: Error finding container 088d61a5706970cfa328fb9eeff8bb8b9d9db1c33edaa6dcdf46d75786fd0ae3: Status 404 returned error can't find the container with id 088d61a5706970cfa328fb9eeff8bb8b9d9db1c33edaa6dcdf46d75786fd0ae3
Sep 09 06:01:53.874174 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.874135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6263b891-ad1a-4c05-9754-5c75cb38b50c-maas-api-tls\") pod \"maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:53.876955 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:53.876927    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6263b891-ad1a-4c05-9754-5c75cb38b50c-maas-api-tls\") pod \"maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:54.155140 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:54.155045    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:54.310954 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:54.310919    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq" event={"ID":"bfe57a46-2294-467f-93a2-d1386a262c69","Type":"ContainerStarted","Data":"64c43228fa96b953683802fcf435666fae09dca50b8e7ccd54794cffc9150e84"}
Sep 09 06:01:54.311139 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:54.310979    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"
Sep 09 06:01:54.311139 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:54.311004    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq" event={"ID":"bfe57a46-2294-467f-93a2-d1386a262c69","Type":"ContainerStarted","Data":"528363aedbaa6b0b925514c8280c6df331a834e6375d9657bb364578ce06d68f"}
Sep 09 06:01:54.312638 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:54.312613    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf" event={"ID":"94d23818-c30b-4b0a-8469-3575669b2bf9","Type":"ContainerStarted","Data":"c62615a329e75a9b969459a8d95557be3f1fd9f8f95811d5dd4576721ec86bf7"}
Sep 09 06:01:54.312749 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:54.312654    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"
Sep 09 06:01:54.312749 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:54.312669    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf" event={"ID":"94d23818-c30b-4b0a-8469-3575669b2bf9","Type":"ContainerStarted","Data":"088d61a5706970cfa328fb9eeff8bb8b9d9db1c33edaa6dcdf46d75786fd0ae3"}
Sep 09 06:01:54.329739 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:54.329684    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq" podStartSLOduration=2.329665114 podStartE2EDuration="2.329665114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:01:54.327710482 +0000 UTC m=+741.544040366" watchObservedRunningTime="2026-09-09 06:01:54.329665114 +0000 UTC m=+741.545994999"
Sep 09 06:01:54.344611 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:54.344553    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf" podStartSLOduration=2.344537107 podStartE2EDuration="2.344537107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:01:54.34237303 +0000 UTC m=+741.558702917" watchObservedRunningTime="2026-09-09 06:01:54.344537107 +0000 UTC m=+741.560867042"
Sep 09 06:01:54.432575 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:54.432546    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"]
Sep 09 06:01:54.433993 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:01:54.433965    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6263b891_ad1a_4c05_9754_5c75cb38b50c.slice/crio-7b94ed9afe1ecfe32626400d789df528380e1313620f96046fb2abcc13402f3b WatchSource:0}: Error finding container 7b94ed9afe1ecfe32626400d789df528380e1313620f96046fb2abcc13402f3b: Status 404 returned error can't find the container with id 7b94ed9afe1ecfe32626400d789df528380e1313620f96046fb2abcc13402f3b
Sep 09 06:01:55.321646 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:55.321604    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5" event={"ID":"6263b891-ad1a-4c05-9754-5c75cb38b50c","Type":"ContainerStarted","Data":"7b94ed9afe1ecfe32626400d789df528380e1313620f96046fb2abcc13402f3b"}
Sep 09 06:01:57.336097 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:57.336065    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5" event={"ID":"6263b891-ad1a-4c05-9754-5c75cb38b50c","Type":"ContainerStarted","Data":"8f7636291c6a3d4ec11f95cbae14aab1e9708f1b9f547ebec762ac687c29e4b5"}
Sep 09 06:01:57.336462 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:57.336121    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:01:57.354196 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:01:57.354145    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5" podStartSLOduration=2.488006117 podStartE2EDuration="4.354127098s" totalImagesPullingTime="1.866120981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:01:57.3530455 +0000 UTC m=+744.569375385" watchObservedRunningTime="2026-09-09 06:01:57.354127098 +0000 UTC m=+744.570456979"
Sep 09 06:02:01.054392 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.054358    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"]
Sep 09 06:02:01.061626 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.061596    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"
Sep 09 06:02:01.066074 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.065792    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c53eb05a\""
Sep 09 06:02:01.066074 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.065871    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c53eb05a-dockercfg-7w7mg\""
Sep 09 06:02:01.072286 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.072254    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"]
Sep 09 06:02:01.121829 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.121792    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"]
Sep 09 06:02:01.125526 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.125480    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"
Sep 09 06:02:01.134036 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.134012    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"]
Sep 09 06:02:01.148565 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.148509    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz7wp\" (UniqueName: \"kubernetes.io/projected/17aae636-f681-4a36-a6bd-050317021b7e-kube-api-access-xz7wp\") pod \"payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp\" (UID: \"17aae636-f681-4a36-a6bd-050317021b7e\") " pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"
Sep 09 06:02:01.148806 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.148572    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17aae636-f681-4a36-a6bd-050317021b7e-plugins-config-volume\") pod \"payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp\" (UID: \"17aae636-f681-4a36-a6bd-050317021b7e\") " pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"
Sep 09 06:02:01.249668 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.249631    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz7wp\" (UniqueName: \"kubernetes.io/projected/17aae636-f681-4a36-a6bd-050317021b7e-kube-api-access-xz7wp\") pod \"payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp\" (UID: \"17aae636-f681-4a36-a6bd-050317021b7e\") " pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"
Sep 09 06:02:01.249872 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.249690    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17aae636-f681-4a36-a6bd-050317021b7e-plugins-config-volume\") pod \"payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp\" (UID: \"17aae636-f681-4a36-a6bd-050317021b7e\") " pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"
Sep 09 06:02:01.249872 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.249776    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbdea98-88ca-4f98-8ab9-2f08744e16ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp\" (UID: \"4fbdea98-88ca-4f98-8ab9-2f08744e16ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"
Sep 09 06:02:01.249872 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.249825    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqh8t\" (UniqueName: \"kubernetes.io/projected/4fbdea98-88ca-4f98-8ab9-2f08744e16ac-kube-api-access-nqh8t\") pod \"payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp\" (UID: \"4fbdea98-88ca-4f98-8ab9-2f08744e16ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"
Sep 09 06:02:01.250474 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.250453    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17aae636-f681-4a36-a6bd-050317021b7e-plugins-config-volume\") pod \"payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp\" (UID: \"17aae636-f681-4a36-a6bd-050317021b7e\") " pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"
Sep 09 06:02:01.262139 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.262102    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz7wp\" (UniqueName: \"kubernetes.io/projected/17aae636-f681-4a36-a6bd-050317021b7e-kube-api-access-xz7wp\") pod \"payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp\" (UID: \"17aae636-f681-4a36-a6bd-050317021b7e\") " pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"
Sep 09 06:02:01.350880 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.350840    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqh8t\" (UniqueName: \"kubernetes.io/projected/4fbdea98-88ca-4f98-8ab9-2f08744e16ac-kube-api-access-nqh8t\") pod \"payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp\" (UID: \"4fbdea98-88ca-4f98-8ab9-2f08744e16ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"
Sep 09 06:02:01.351085 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.350996    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbdea98-88ca-4f98-8ab9-2f08744e16ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp\" (UID: \"4fbdea98-88ca-4f98-8ab9-2f08744e16ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"
Sep 09 06:02:01.351931 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.351905    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbdea98-88ca-4f98-8ab9-2f08744e16ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp\" (UID: \"4fbdea98-88ca-4f98-8ab9-2f08744e16ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"
Sep 09 06:02:01.364139 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.364111    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqh8t\" (UniqueName: \"kubernetes.io/projected/4fbdea98-88ca-4f98-8ab9-2f08744e16ac-kube-api-access-nqh8t\") pod \"payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp\" (UID: \"4fbdea98-88ca-4f98-8ab9-2f08744e16ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"
Sep 09 06:02:01.377735 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.377710    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"
Sep 09 06:02:01.439406 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.439366    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"
Sep 09 06:02:01.665533 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.665508    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"]
Sep 09 06:02:01.667196 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:02:01.667167    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17aae636_f681_4a36_a6bd_050317021b7e.slice/crio-7720bbdea1dd9fd456e350e77cf9ef12361985e10842158bf08180ba63a04317 WatchSource:0}: Error finding container 7720bbdea1dd9fd456e350e77cf9ef12361985e10842158bf08180ba63a04317: Status 404 returned error can't find the container with id 7720bbdea1dd9fd456e350e77cf9ef12361985e10842158bf08180ba63a04317
Sep 09 06:02:01.720437 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:01.720410    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"]
Sep 09 06:02:01.721240 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:02:01.721217    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbdea98_88ca_4f98_8ab9_2f08744e16ac.slice/crio-077da7271a94d27ea398763ad1b99645837112dd6f48dfbff04d860d5d8959a5 WatchSource:0}: Error finding container 077da7271a94d27ea398763ad1b99645837112dd6f48dfbff04d860d5d8959a5: Status 404 returned error can't find the container with id 077da7271a94d27ea398763ad1b99645837112dd6f48dfbff04d860d5d8959a5
Sep 09 06:02:02.369121 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:02.369082    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp" event={"ID":"4fbdea98-88ca-4f98-8ab9-2f08744e16ac","Type":"ContainerStarted","Data":"aec16b415a88bf23a309793a4cfdaf2f51bca2638a00c00655256c5191b400fe"}
Sep 09 06:02:02.369639 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:02.369144    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp" event={"ID":"4fbdea98-88ca-4f98-8ab9-2f08744e16ac","Type":"ContainerStarted","Data":"077da7271a94d27ea398763ad1b99645837112dd6f48dfbff04d860d5d8959a5"}
Sep 09 06:02:02.369639 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:02.369175    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"
Sep 09 06:02:02.370811 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:02.370782    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp" event={"ID":"17aae636-f681-4a36-a6bd-050317021b7e","Type":"ContainerStarted","Data":"e53af76e74de29228ee1696a296f1fd677af7f0eb28f0e5d47d42af976219f09"}
Sep 09 06:02:02.370928 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:02.370827    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp" event={"ID":"17aae636-f681-4a36-a6bd-050317021b7e","Type":"ContainerStarted","Data":"7720bbdea1dd9fd456e350e77cf9ef12361985e10842158bf08180ba63a04317"}
Sep 09 06:02:02.370928 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:02.370907    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"
Sep 09 06:02:02.387301 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:02.387253    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp" podStartSLOduration=1.387237731 podStartE2EDuration="1.387237731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:02:02.38459594 +0000 UTC m=+749.600925826" watchObservedRunningTime="2026-09-09 06:02:02.387237731 +0000 UTC m=+749.603567615"
Sep 09 06:02:02.404098 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:02.404047    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp" podStartSLOduration=1.404030407 podStartE2EDuration="1.404030407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:02:02.402000032 +0000 UTC m=+749.618329917" watchObservedRunningTime="2026-09-09 06:02:02.404030407 +0000 UTC m=+749.620360290"
Sep 09 06:02:03.347545 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:03.347516    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:02:05.324105 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:05.324076    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"
Sep 09 06:02:05.324613 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:05.324286    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"
Sep 09 06:02:05.699617 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:05.699542    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"]
Sep 09 06:02:05.703444 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:05.703423    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"
Sep 09 06:02:05.705876 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:05.705849    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-7a357a\""
Sep 09 06:02:05.705969 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:05.705917    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-7a357a-dockercfg-jxtkf\""
Sep 09 06:02:05.715308 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:05.715281    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"]
Sep 09 06:02:05.796998 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:05.796937    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910a5724-7ee3-4490-b58b-cc590f14ecf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f\" (UID: \"910a5724-7ee3-4490-b58b-cc590f14ecf3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"
Sep 09 06:02:05.797162 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:05.797073    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhj4\" (UniqueName: \"kubernetes.io/projected/910a5724-7ee3-4490-b58b-cc590f14ecf3-kube-api-access-tqhj4\") pod \"payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f\" (UID: \"910a5724-7ee3-4490-b58b-cc590f14ecf3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"
Sep 09 06:02:05.898503 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:05.898454    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910a5724-7ee3-4490-b58b-cc590f14ecf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f\" (UID: \"910a5724-7ee3-4490-b58b-cc590f14ecf3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"
Sep 09 06:02:05.898679 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:05.898551    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhj4\" (UniqueName: \"kubernetes.io/projected/910a5724-7ee3-4490-b58b-cc590f14ecf3-kube-api-access-tqhj4\") pod \"payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f\" (UID: \"910a5724-7ee3-4490-b58b-cc590f14ecf3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"
Sep 09 06:02:05.899159 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:05.899138    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910a5724-7ee3-4490-b58b-cc590f14ecf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f\" (UID: \"910a5724-7ee3-4490-b58b-cc590f14ecf3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"
Sep 09 06:02:05.910967 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:05.910934    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhj4\" (UniqueName: \"kubernetes.io/projected/910a5724-7ee3-4490-b58b-cc590f14ecf3-kube-api-access-tqhj4\") pod \"payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f\" (UID: \"910a5724-7ee3-4490-b58b-cc590f14ecf3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"
Sep 09 06:02:06.015399 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:06.015318    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"
Sep 09 06:02:06.290298 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:06.290267    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"]
Sep 09 06:02:06.292001 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:02:06.291975    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910a5724_7ee3_4490_b58b_cc590f14ecf3.slice/crio-8d7c83a45bc9a2f89cec54d16b2c8872f4e3eee442865c1aa863c6908910a43b WatchSource:0}: Error finding container 8d7c83a45bc9a2f89cec54d16b2c8872f4e3eee442865c1aa863c6908910a43b: Status 404 returned error can't find the container with id 8d7c83a45bc9a2f89cec54d16b2c8872f4e3eee442865c1aa863c6908910a43b
Sep 09 06:02:06.409811 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:06.409770    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f" event={"ID":"910a5724-7ee3-4490-b58b-cc590f14ecf3","Type":"ContainerStarted","Data":"355a64036b6ff11841cdeced272ea17dcedf5d7001f80e34eaa68089e3d7ed88"}
Sep 09 06:02:06.410177 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:06.409835    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f" event={"ID":"910a5724-7ee3-4490-b58b-cc590f14ecf3","Type":"ContainerStarted","Data":"8d7c83a45bc9a2f89cec54d16b2c8872f4e3eee442865c1aa863c6908910a43b"}
Sep 09 06:02:06.410177 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:06.409949    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"
Sep 09 06:02:06.426379 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:06.426312    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f" podStartSLOduration=1.426295616 podStartE2EDuration="1.426295616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:02:06.424911097 +0000 UTC m=+753.641241001" watchObservedRunningTime="2026-09-09 06:02:06.426295616 +0000 UTC m=+753.642625509"
Sep 09 06:02:13.384443 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:13.384412    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"
Sep 09 06:02:13.384845 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:13.384521    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"
Sep 09 06:02:17.420876 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:17.420844    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"
Sep 09 06:02:24.811095 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:24.811059    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg"]
Sep 09 06:02:24.816101 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:24.816082    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg"
Sep 09 06:02:24.818758 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:24.818732    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:02:24.824885 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:24.824856    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg"]
Sep 09 06:02:24.881853 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:24.881823    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12ea7b2-dc1e-48ec-b434-65b98816816e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg\" (UID: \"e12ea7b2-dc1e-48ec-b434-65b98816816e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg"
Sep 09 06:02:24.983161 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:24.983122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12ea7b2-dc1e-48ec-b434-65b98816816e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg\" (UID: \"e12ea7b2-dc1e-48ec-b434-65b98816816e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg"
Sep 09 06:02:24.983768 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:24.983749    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12ea7b2-dc1e-48ec-b434-65b98816816e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg\" (UID: \"e12ea7b2-dc1e-48ec-b434-65b98816816e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg"
Sep 09 06:02:25.128405 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:25.128367    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg"
Sep 09 06:02:25.409945 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:25.409920    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg"]
Sep 09 06:02:25.411204 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:02:25.411182    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12ea7b2_dc1e_48ec_b434_65b98816816e.slice/crio-817391f28a8106f25b85031b229495a5b85175a1e0662cac0d6a0467868e1557 WatchSource:0}: Error finding container 817391f28a8106f25b85031b229495a5b85175a1e0662cac0d6a0467868e1557: Status 404 returned error can't find the container with id 817391f28a8106f25b85031b229495a5b85175a1e0662cac0d6a0467868e1557
Sep 09 06:02:25.532156 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:25.532122    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg" event={"ID":"e12ea7b2-dc1e-48ec-b434-65b98816816e","Type":"ContainerStarted","Data":"817391f28a8106f25b85031b229495a5b85175a1e0662cac0d6a0467868e1557"}
Sep 09 06:02:28.557334 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:28.557298    2564 generic.go:356] "Generic (PLEG): container finished" podID="e12ea7b2-dc1e-48ec-b434-65b98816816e" containerID="2c3779946ac15eda8934b418f233de11da35c257b669640afefd1ed617925b0b" exitCode=0
Sep 09 06:02:28.557726 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:28.557396    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg" event={"ID":"e12ea7b2-dc1e-48ec-b434-65b98816816e","Type":"ContainerDied","Data":"2c3779946ac15eda8934b418f233de11da35c257b669640afefd1ed617925b0b"}
Sep 09 06:02:29.710965 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:29.710943    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg"
Sep 09 06:02:29.733639 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:29.733606    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12ea7b2-dc1e-48ec-b434-65b98816816e-maas-api-service-ca\") pod \"e12ea7b2-dc1e-48ec-b434-65b98816816e\" (UID: \"e12ea7b2-dc1e-48ec-b434-65b98816816e\") "
Sep 09 06:02:29.734020 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:29.733995    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12ea7b2-dc1e-48ec-b434-65b98816816e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e12ea7b2-dc1e-48ec-b434-65b98816816e" (UID: "e12ea7b2-dc1e-48ec-b434-65b98816816e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:02:29.835361 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:29.835329    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12ea7b2-dc1e-48ec-b434-65b98816816e-maas-api-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:02:30.572782 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:30.572752    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg"
Sep 09 06:02:30.572962 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:30.572808    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg" event={"ID":"e12ea7b2-dc1e-48ec-b434-65b98816816e","Type":"ContainerDied","Data":"817391f28a8106f25b85031b229495a5b85175a1e0662cac0d6a0467868e1557"}
Sep 09 06:02:30.572962 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:30.572865    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="817391f28a8106f25b85031b229495a5b85175a1e0662cac0d6a0467868e1557"
Sep 09 06:02:41.223432 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.222516    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"]
Sep 09 06:02:41.224571 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.224369    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp" podUID="17aae636-f681-4a36-a6bd-050317021b7e" containerName="payload-processing" containerID="cri-o://e53af76e74de29228ee1696a296f1fd677af7f0eb28f0e5d47d42af976219f09" gracePeriod=30
Sep 09 06:02:41.233397 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.233287    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"]
Sep 09 06:02:41.233604 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.233559    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp" podUID="4fbdea98-88ca-4f98-8ab9-2f08744e16ac" containerName="payload-pre-processing" containerID="cri-o://aec16b415a88bf23a309793a4cfdaf2f51bca2638a00c00655256c5191b400fe" gracePeriod=30
Sep 09 06:02:41.373476 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.373432    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg"]
Sep 09 06:02:41.377116 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.377086    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53eb05a-m6zmg"]
Sep 09 06:02:41.544023 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.543989    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"
Sep 09 06:02:41.547461 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.547437    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"
Sep 09 06:02:41.644192 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.644155    2564 generic.go:356] "Generic (PLEG): container finished" podID="17aae636-f681-4a36-a6bd-050317021b7e" containerID="e53af76e74de29228ee1696a296f1fd677af7f0eb28f0e5d47d42af976219f09" exitCode=0
Sep 09 06:02:41.644378 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.644193    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp" event={"ID":"17aae636-f681-4a36-a6bd-050317021b7e","Type":"ContainerDied","Data":"e53af76e74de29228ee1696a296f1fd677af7f0eb28f0e5d47d42af976219f09"}
Sep 09 06:02:41.644378 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.644211    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"
Sep 09 06:02:41.644378 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.644263    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp" event={"ID":"17aae636-f681-4a36-a6bd-050317021b7e","Type":"ContainerDied","Data":"7720bbdea1dd9fd456e350e77cf9ef12361985e10842158bf08180ba63a04317"}
Sep 09 06:02:41.644378 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.644295    2564 scope.go:117] "RemoveContainer" containerID="e53af76e74de29228ee1696a296f1fd677af7f0eb28f0e5d47d42af976219f09"
Sep 09 06:02:41.645813 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.645792    2564 generic.go:356] "Generic (PLEG): container finished" podID="4fbdea98-88ca-4f98-8ab9-2f08744e16ac" containerID="aec16b415a88bf23a309793a4cfdaf2f51bca2638a00c00655256c5191b400fe" exitCode=0
Sep 09 06:02:41.645936 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.645837    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"
Sep 09 06:02:41.645936 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.645879    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp" event={"ID":"4fbdea98-88ca-4f98-8ab9-2f08744e16ac","Type":"ContainerDied","Data":"aec16b415a88bf23a309793a4cfdaf2f51bca2638a00c00655256c5191b400fe"}
Sep 09 06:02:41.645936 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.645927    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp" event={"ID":"4fbdea98-88ca-4f98-8ab9-2f08744e16ac","Type":"ContainerDied","Data":"077da7271a94d27ea398763ad1b99645837112dd6f48dfbff04d860d5d8959a5"}
Sep 09 06:02:41.652464 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.652441    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz7wp\" (UniqueName: \"kubernetes.io/projected/17aae636-f681-4a36-a6bd-050317021b7e-kube-api-access-xz7wp\") pod \"17aae636-f681-4a36-a6bd-050317021b7e\" (UID: \"17aae636-f681-4a36-a6bd-050317021b7e\") "
Sep 09 06:02:41.652593 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.652519    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17aae636-f681-4a36-a6bd-050317021b7e-plugins-config-volume\") pod \"17aae636-f681-4a36-a6bd-050317021b7e\" (UID: \"17aae636-f681-4a36-a6bd-050317021b7e\") "
Sep 09 06:02:41.652665 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.652623    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqh8t\" (UniqueName: \"kubernetes.io/projected/4fbdea98-88ca-4f98-8ab9-2f08744e16ac-kube-api-access-nqh8t\") pod \"4fbdea98-88ca-4f98-8ab9-2f08744e16ac\" (UID: \"4fbdea98-88ca-4f98-8ab9-2f08744e16ac\") "
Sep 09 06:02:41.652721 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.652710    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbdea98-88ca-4f98-8ab9-2f08744e16ac-plugins-config-volume\") pod \"4fbdea98-88ca-4f98-8ab9-2f08744e16ac\" (UID: \"4fbdea98-88ca-4f98-8ab9-2f08744e16ac\") "
Sep 09 06:02:41.652913 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.652891    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17aae636-f681-4a36-a6bd-050317021b7e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17aae636-f681-4a36-a6bd-050317021b7e" (UID: "17aae636-f681-4a36-a6bd-050317021b7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:02:41.653221 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.653194    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbdea98-88ca-4f98-8ab9-2f08744e16ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4fbdea98-88ca-4f98-8ab9-2f08744e16ac" (UID: "4fbdea98-88ca-4f98-8ab9-2f08744e16ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:02:41.654013 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.653996    2564 scope.go:117] "RemoveContainer" containerID="e53af76e74de29228ee1696a296f1fd677af7f0eb28f0e5d47d42af976219f09"
Sep 09 06:02:41.654317 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:02:41.654298    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e53af76e74de29228ee1696a296f1fd677af7f0eb28f0e5d47d42af976219f09\": container with ID starting with e53af76e74de29228ee1696a296f1fd677af7f0eb28f0e5d47d42af976219f09 not found: ID does not exist" containerID="e53af76e74de29228ee1696a296f1fd677af7f0eb28f0e5d47d42af976219f09"
Sep 09 06:02:41.654398 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.654322    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e53af76e74de29228ee1696a296f1fd677af7f0eb28f0e5d47d42af976219f09"} err="failed to get container status \"e53af76e74de29228ee1696a296f1fd677af7f0eb28f0e5d47d42af976219f09\": rpc error: code = NotFound desc = could not find container \"e53af76e74de29228ee1696a296f1fd677af7f0eb28f0e5d47d42af976219f09\": container with ID starting with e53af76e74de29228ee1696a296f1fd677af7f0eb28f0e5d47d42af976219f09 not found: ID does not exist"
Sep 09 06:02:41.654398 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.654338    2564 scope.go:117] "RemoveContainer" containerID="aec16b415a88bf23a309793a4cfdaf2f51bca2638a00c00655256c5191b400fe"
Sep 09 06:02:41.655174 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.655149    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17aae636-f681-4a36-a6bd-050317021b7e-kube-api-access-xz7wp" (OuterVolumeSpecName: "kube-api-access-xz7wp") pod "17aae636-f681-4a36-a6bd-050317021b7e" (UID: "17aae636-f681-4a36-a6bd-050317021b7e"). InnerVolumeSpecName "kube-api-access-xz7wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:02:41.655253 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.655178    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbdea98-88ca-4f98-8ab9-2f08744e16ac-kube-api-access-nqh8t" (OuterVolumeSpecName: "kube-api-access-nqh8t") pod "4fbdea98-88ca-4f98-8ab9-2f08744e16ac" (UID: "4fbdea98-88ca-4f98-8ab9-2f08744e16ac"). InnerVolumeSpecName "kube-api-access-nqh8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:02:41.663332 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.663312    2564 scope.go:117] "RemoveContainer" containerID="aec16b415a88bf23a309793a4cfdaf2f51bca2638a00c00655256c5191b400fe"
Sep 09 06:02:41.663700 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:02:41.663678    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec16b415a88bf23a309793a4cfdaf2f51bca2638a00c00655256c5191b400fe\": container with ID starting with aec16b415a88bf23a309793a4cfdaf2f51bca2638a00c00655256c5191b400fe not found: ID does not exist" containerID="aec16b415a88bf23a309793a4cfdaf2f51bca2638a00c00655256c5191b400fe"
Sep 09 06:02:41.663806 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.663708    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec16b415a88bf23a309793a4cfdaf2f51bca2638a00c00655256c5191b400fe"} err="failed to get container status \"aec16b415a88bf23a309793a4cfdaf2f51bca2638a00c00655256c5191b400fe\": rpc error: code = NotFound desc = could not find container \"aec16b415a88bf23a309793a4cfdaf2f51bca2638a00c00655256c5191b400fe\": container with ID starting with aec16b415a88bf23a309793a4cfdaf2f51bca2638a00c00655256c5191b400fe not found: ID does not exist"
Sep 09 06:02:41.754042 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.754006    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbdea98-88ca-4f98-8ab9-2f08744e16ac-plugins-config-volume\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:02:41.754042 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.754035    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz7wp\" (UniqueName: \"kubernetes.io/projected/17aae636-f681-4a36-a6bd-050317021b7e-kube-api-access-xz7wp\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:02:41.754042 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.754045    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17aae636-f681-4a36-a6bd-050317021b7e-plugins-config-volume\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:02:41.754281 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.754053    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqh8t\" (UniqueName: \"kubernetes.io/projected/4fbdea98-88ca-4f98-8ab9-2f08744e16ac-kube-api-access-nqh8t\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:02:41.972699 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.972667    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"]
Sep 09 06:02:41.977234 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:41.977207    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c53eb05a-85c9f88b54-nldvp"]
Sep 09 06:02:42.000436 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.000387    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"]
Sep 09 06:02:42.002973 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.002954    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c53eb05a-c95fbb747-ct6dp"]
Sep 09 06:02:42.543451 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.543365    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"]
Sep 09 06:02:42.544568 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.544540    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e12ea7b2-dc1e-48ec-b434-65b98816816e" containerName="revoke-keys"
Sep 09 06:02:42.544568 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.544568    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e12ea7b2-dc1e-48ec-b434-65b98816816e" containerName="revoke-keys"
Sep 09 06:02:42.544727 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.544599    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17aae636-f681-4a36-a6bd-050317021b7e" containerName="payload-processing"
Sep 09 06:02:42.544727 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.544608    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="17aae636-f681-4a36-a6bd-050317021b7e" containerName="payload-processing"
Sep 09 06:02:42.544727 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.544634    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fbdea98-88ca-4f98-8ab9-2f08744e16ac" containerName="payload-pre-processing"
Sep 09 06:02:42.544727 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.544643    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fbdea98-88ca-4f98-8ab9-2f08744e16ac" containerName="payload-pre-processing"
Sep 09 06:02:42.544859 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.544764    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="17aae636-f681-4a36-a6bd-050317021b7e" containerName="payload-processing"
Sep 09 06:02:42.544859 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.544776    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fbdea98-88ca-4f98-8ab9-2f08744e16ac" containerName="payload-pre-processing"
Sep 09 06:02:42.544859 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.544785    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e12ea7b2-dc1e-48ec-b434-65b98816816e" containerName="revoke-keys"
Sep 09 06:02:42.549610 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.549593    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.553191 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.553173    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e7b233dd-gw-openshift-default-dockercfg-vsmzt\""
Sep 09 06:02:42.563102 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.563078    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"]
Sep 09 06:02:42.663852 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.663823    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76716474-bb47-4751-81ac-b09ff1915b00-istiod-ca-cert\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.664024 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.663859    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-credential-socket\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.664024 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.663887    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-istio-envoy\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.664024 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.663934    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76716474-bb47-4751-81ac-b09ff1915b00-istio-token\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.664134 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.664024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-istio-data\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.664134 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.664072    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76716474-bb47-4751-81ac-b09ff1915b00-istio-podinfo\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.664134 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.664102    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bqh\" (UniqueName: \"kubernetes.io/projected/76716474-bb47-4751-81ac-b09ff1915b00-kube-api-access-n4bqh\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.664230 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.664181    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-workload-socket\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.664230 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.664224    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-workload-certs\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.765252 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.765215    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-istio-data\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.765252 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.765257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76716474-bb47-4751-81ac-b09ff1915b00-istio-podinfo\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.765564 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.765285    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bqh\" (UniqueName: \"kubernetes.io/projected/76716474-bb47-4751-81ac-b09ff1915b00-kube-api-access-n4bqh\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.765564 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.765343    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-workload-socket\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.765564 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.765387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-workload-certs\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.765564 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.765454    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76716474-bb47-4751-81ac-b09ff1915b00-istiod-ca-cert\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.765564 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.765488    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-credential-socket\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.765564 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.765548    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-istio-envoy\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.765889 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.765575    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76716474-bb47-4751-81ac-b09ff1915b00-istio-token\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.765889 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.765761    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-istio-data\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.765889 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.765797    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-workload-socket\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.765889 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.765827    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-workload-certs\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.766056 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.765993    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-credential-socket\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.766286 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.766266    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76716474-bb47-4751-81ac-b09ff1915b00-istiod-ca-cert\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.767900 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.767869    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76716474-bb47-4751-81ac-b09ff1915b00-istio-podinfo\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.768125 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.768108    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-istio-envoy\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.772675 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.772650    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76716474-bb47-4751-81ac-b09ff1915b00-istio-token\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.772851 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.772833    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bqh\" (UniqueName: \"kubernetes.io/projected/76716474-bb47-4751-81ac-b09ff1915b00-kube-api-access-n4bqh\") pod \"e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") " pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:42.860319 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:42.860280    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:43.140838 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:43.140807    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"]
Sep 09 06:02:43.141936 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:02:43.141910    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76716474_bb47_4751_81ac_b09ff1915b00.slice/crio-94419cc28e124db6a6f32d85dd3a8a42b4608c5ac5208b69e8d23b0aeb5d7688 WatchSource:0}: Error finding container 94419cc28e124db6a6f32d85dd3a8a42b4608c5ac5208b69e8d23b0aeb5d7688: Status 404 returned error can't find the container with id 94419cc28e124db6a6f32d85dd3a8a42b4608c5ac5208b69e8d23b0aeb5d7688
Sep 09 06:02:43.144227 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:43.144201    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:02:43.144300 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:43.144258    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:02:43.144300 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:43.144285    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:02:43.304635 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:43.304602    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17aae636-f681-4a36-a6bd-050317021b7e" path="/var/lib/kubelet/pods/17aae636-f681-4a36-a6bd-050317021b7e/volumes"
Sep 09 06:02:43.305013 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:43.305000    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fbdea98-88ca-4f98-8ab9-2f08744e16ac" path="/var/lib/kubelet/pods/4fbdea98-88ca-4f98-8ab9-2f08744e16ac/volumes"
Sep 09 06:02:43.305301 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:43.305281    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e12ea7b2-dc1e-48ec-b434-65b98816816e" path="/var/lib/kubelet/pods/e12ea7b2-dc1e-48ec-b434-65b98816816e/volumes"
Sep 09 06:02:43.664134 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:43.664090    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj" event={"ID":"76716474-bb47-4751-81ac-b09ff1915b00","Type":"ContainerStarted","Data":"48a679ec70a61e682138249c9a26532166b0382b466e0cb397c554a8555b977f"}
Sep 09 06:02:43.664526 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:43.664149    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj" event={"ID":"76716474-bb47-4751-81ac-b09ff1915b00","Type":"ContainerStarted","Data":"94419cc28e124db6a6f32d85dd3a8a42b4608c5ac5208b69e8d23b0aeb5d7688"}
Sep 09 06:02:43.689352 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:43.689284    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj" podStartSLOduration=1.68926396 podStartE2EDuration="1.68926396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:02:43.684391674 +0000 UTC m=+790.900721568" watchObservedRunningTime="2026-09-09 06:02:43.68926396 +0000 UTC m=+790.905593844"
Sep 09 06:02:43.861050 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:43.861000    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:44.866173 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:44.866137    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:45.502118 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.502078    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"]
Sep 09 06:02:45.508055 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.508019    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:45.510408 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.510384    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e7b233dd-serving-cert\""
Sep 09 06:02:45.513814 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.513788    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"]
Sep 09 06:02:45.594314 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.594276    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7049f435-8185-4690-9568-14d9b339013a-metrics-certs\") pod \"maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:45.594314 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.594313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7049f435-8185-4690-9568-14d9b339013a-maas-api-tls\") pod \"maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:45.594532 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.594367    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh2rb\" (UniqueName: \"kubernetes.io/projected/7049f435-8185-4690-9568-14d9b339013a-kube-api-access-bh2rb\") pod \"maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:45.594532 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.594414    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7049f435-8185-4690-9568-14d9b339013a-openshift-service-ca\") pod \"maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:45.676485 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.676452    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:45.677439 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.677419    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:02:45.695124 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.695095    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7049f435-8185-4690-9568-14d9b339013a-metrics-certs\") pod \"maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:45.695260 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.695140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7049f435-8185-4690-9568-14d9b339013a-maas-api-tls\") pod \"maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:45.695260 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.695215    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh2rb\" (UniqueName: \"kubernetes.io/projected/7049f435-8185-4690-9568-14d9b339013a-kube-api-access-bh2rb\") pod \"maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:45.695260 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.695249    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7049f435-8185-4690-9568-14d9b339013a-openshift-service-ca\") pod \"maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:45.695435 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:02:45.695413    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-serving-cert: secret "maas-api-e2e-migrate-e7b233dd-serving-cert" not found
Sep 09 06:02:45.695673 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:02:45.695648    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7049f435-8185-4690-9568-14d9b339013a-maas-api-tls podName:7049f435-8185-4690-9568-14d9b339013a nodeName:}" failed. No retries permitted until 2026-09-09 06:02:46.195624169 +0000 UTC m=+793.411954036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7049f435-8185-4690-9568-14d9b339013a-maas-api-tls") pod "maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475" (UID: "7049f435-8185-4690-9568-14d9b339013a") : secret "maas-api-e2e-migrate-e7b233dd-serving-cert" not found
Sep 09 06:02:45.696474 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.696451    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7049f435-8185-4690-9568-14d9b339013a-openshift-service-ca\") pod \"maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:45.698650 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.698629    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7049f435-8185-4690-9568-14d9b339013a-metrics-certs\") pod \"maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:45.709735 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:45.709711    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh2rb\" (UniqueName: \"kubernetes.io/projected/7049f435-8185-4690-9568-14d9b339013a-kube-api-access-bh2rb\") pod \"maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:46.199347 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:46.199309    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7049f435-8185-4690-9568-14d9b339013a-maas-api-tls\") pod \"maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:46.201772 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:46.201749    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7049f435-8185-4690-9568-14d9b339013a-maas-api-tls\") pod \"maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:46.423146 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:46.423108    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:46.703890 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:46.703862    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"]
Sep 09 06:02:46.705149 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:02:46.705126    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7049f435_8185_4690_9568_14d9b339013a.slice/crio-2ac3826a2746cfa7790e9ee845c7734650d78a21b6e8c164b020f82ec6de5f4f WatchSource:0}: Error finding container 2ac3826a2746cfa7790e9ee845c7734650d78a21b6e8c164b020f82ec6de5f4f: Status 404 returned error can't find the container with id 2ac3826a2746cfa7790e9ee845c7734650d78a21b6e8c164b020f82ec6de5f4f
Sep 09 06:02:47.690593 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:47.690559    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475" event={"ID":"7049f435-8185-4690-9568-14d9b339013a","Type":"ContainerStarted","Data":"a0c81a1c8f727782d51b0613a01604c698f72468344deeb1107cd62c0329f916"}
Sep 09 06:02:47.691020 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:47.690618    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475" event={"ID":"7049f435-8185-4690-9568-14d9b339013a","Type":"ContainerStarted","Data":"2ac3826a2746cfa7790e9ee845c7734650d78a21b6e8c164b020f82ec6de5f4f"}
Sep 09 06:02:47.691020 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:47.690649    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:02:47.717511 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:47.711750    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475" podStartSLOduration=2.395009982 podStartE2EDuration="2.71172197s" totalImagesPullingTime="316.711988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:02:47.708439451 +0000 UTC m=+794.924769326" watchObservedRunningTime="2026-09-09 06:02:47.71172197 +0000 UTC m=+794.928051854"
Sep 09 06:02:53.701926 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:02:53.701894    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:03:01.718678 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:01.718635    2564 patch_prober.go:28] interesting pod/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 06:03:01.719104 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:01.718716    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj" podUID="76716474-bb47-4751-81ac-b09ff1915b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 06:03:04.811575 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:04.811543    2564 generic.go:356] "Generic (PLEG): container finished" podID="76716474-bb47-4751-81ac-b09ff1915b00" containerID="48a679ec70a61e682138249c9a26532166b0382b466e0cb397c554a8555b977f" exitCode=0
Sep 09 06:03:04.811976 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:04.811541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj" event={"ID":"76716474-bb47-4751-81ac-b09ff1915b00","Type":"ContainerDied","Data":"48a679ec70a61e682138249c9a26532166b0382b466e0cb397c554a8555b977f"}
Sep 09 06:03:04.812037 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:04.812025    2564 scope.go:117] "RemoveContainer" containerID="48a679ec70a61e682138249c9a26532166b0382b466e0cb397c554a8555b977f"
Sep 09 06:03:04.813264 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:04.813238    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:03:04.813319 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:04.813301    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:03:04.813354 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:04.813333    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:03:05.819840 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:05.819802    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj" event={"ID":"76716474-bb47-4751-81ac-b09ff1915b00","Type":"ContainerStarted","Data":"fb185b4619d32782d6896544019b5ccf57ed2fcdda81da498a107f3639ab2a3a"}
Sep 09 06:03:05.861479 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:05.861444    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:03:05.861769 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:05.861746    2564 patch_prober.go:28] interesting pod/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 09 06:03:05.861817 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:05.861797    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj" podUID="76716474-bb47-4751-81ac-b09ff1915b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 09 06:03:06.861017 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:06.860968    2564 patch_prober.go:28] interesting pod/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 09 06:03:06.861511 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:06.861040    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj" podUID="76716474-bb47-4751-81ac-b09ff1915b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 09 06:03:07.861277 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:07.860827    2564 patch_prober.go:28] interesting pod/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 09 06:03:07.861277 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:07.860903    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj" podUID="76716474-bb47-4751-81ac-b09ff1915b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 09 06:03:08.861576 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:08.861539    2564 patch_prober.go:28] interesting pod/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 09 06:03:08.862147 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:08.862110    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj" podUID="76716474-bb47-4751-81ac-b09ff1915b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 09 06:03:09.861293 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:09.861254    2564 patch_prober.go:28] interesting pod/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 09 06:03:09.861648 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:09.861616    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj" podUID="76716474-bb47-4751-81ac-b09ff1915b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 09 06:03:10.215409 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:10.215315    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8"]
Sep 09 06:03:10.219003 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:10.218985    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8"
Sep 09 06:03:10.221205 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:10.221185    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:03:10.225871 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:10.225847    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8"]
Sep 09 06:03:10.334884 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:10.334841    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782df8a5-b0b1-4beb-926f-ad47b662efec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8\" (UID: \"782df8a5-b0b1-4beb-926f-ad47b662efec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8"
Sep 09 06:03:10.436236 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:10.436205    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782df8a5-b0b1-4beb-926f-ad47b662efec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8\" (UID: \"782df8a5-b0b1-4beb-926f-ad47b662efec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8"
Sep 09 06:03:10.436807 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:10.436789    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782df8a5-b0b1-4beb-926f-ad47b662efec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8\" (UID: \"782df8a5-b0b1-4beb-926f-ad47b662efec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8"
Sep 09 06:03:10.531398 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:10.531306    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8"
Sep 09 06:03:10.823705 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:10.823668    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8"]
Sep 09 06:03:10.824548 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:03:10.824516    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782df8a5_b0b1_4beb_926f_ad47b662efec.slice/crio-fa010f057eeefd6e5e8d205d994d3e01fdccd582709f689818e5a5908ba2c34f WatchSource:0}: Error finding container fa010f057eeefd6e5e8d205d994d3e01fdccd582709f689818e5a5908ba2c34f: Status 404 returned error can't find the container with id fa010f057eeefd6e5e8d205d994d3e01fdccd582709f689818e5a5908ba2c34f
Sep 09 06:03:10.860525 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:10.859972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8" event={"ID":"782df8a5-b0b1-4beb-926f-ad47b662efec","Type":"ContainerStarted","Data":"fa010f057eeefd6e5e8d205d994d3e01fdccd582709f689818e5a5908ba2c34f"}
Sep 09 06:03:10.864519 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:10.861479    2564 patch_prober.go:28] interesting pod/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 09 06:03:10.864519 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:10.861544    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj" podUID="76716474-bb47-4751-81ac-b09ff1915b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 09 06:03:11.868213 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:11.868174    2564 generic.go:356] "Generic (PLEG): container finished" podID="782df8a5-b0b1-4beb-926f-ad47b662efec" containerID="2f3556c19175a64fd979e7a2d65dded33e9da503fe586355d6dfaadfc1c9498c" exitCode=0
Sep 09 06:03:11.868607 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:11.868212    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8" event={"ID":"782df8a5-b0b1-4beb-926f-ad47b662efec","Type":"ContainerDied","Data":"2f3556c19175a64fd979e7a2d65dded33e9da503fe586355d6dfaadfc1c9498c"}
Sep 09 06:03:11.954696 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:11.954652    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:03:11.955187 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:11.955159    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:03:11.956136 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:11.956115    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:03:13.015929 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:13.015904    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8"
Sep 09 06:03:13.066092 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:13.066064    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782df8a5-b0b1-4beb-926f-ad47b662efec-maas-api-service-ca\") pod \"782df8a5-b0b1-4beb-926f-ad47b662efec\" (UID: \"782df8a5-b0b1-4beb-926f-ad47b662efec\") "
Sep 09 06:03:13.066399 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:13.066377    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782df8a5-b0b1-4beb-926f-ad47b662efec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "782df8a5-b0b1-4beb-926f-ad47b662efec" (UID: "782df8a5-b0b1-4beb-926f-ad47b662efec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:03:13.167624 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:13.167531    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782df8a5-b0b1-4beb-926f-ad47b662efec-maas-api-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:13.884563 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:13.884527    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8" event={"ID":"782df8a5-b0b1-4beb-926f-ad47b662efec","Type":"ContainerDied","Data":"fa010f057eeefd6e5e8d205d994d3e01fdccd582709f689818e5a5908ba2c34f"}
Sep 09 06:03:13.884754 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:13.884585    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa010f057eeefd6e5e8d205d994d3e01fdccd582709f689818e5a5908ba2c34f"
Sep 09 06:03:13.884754 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:13.884538    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8"
Sep 09 06:03:15.940013 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:15.939979    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"]
Sep 09 06:03:15.940422 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:15.940229    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475" podUID="7049f435-8185-4690-9568-14d9b339013a" containerName="maas-api" containerID="cri-o://a0c81a1c8f727782d51b0613a01604c698f72468344deeb1107cd62c0329f916" gracePeriod=30
Sep 09 06:03:16.175387 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.175353    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8"]
Sep 09 06:03:16.178848 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.178818    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7b233dd-xqhp8"]
Sep 09 06:03:16.203992 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.203938    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:03:16.302793 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.302757    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7049f435-8185-4690-9568-14d9b339013a-openshift-service-ca\") pod \"7049f435-8185-4690-9568-14d9b339013a\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") "
Sep 09 06:03:16.302949 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.302814    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh2rb\" (UniqueName: \"kubernetes.io/projected/7049f435-8185-4690-9568-14d9b339013a-kube-api-access-bh2rb\") pod \"7049f435-8185-4690-9568-14d9b339013a\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") "
Sep 09 06:03:16.302949 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.302933    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7049f435-8185-4690-9568-14d9b339013a-metrics-certs\") pod \"7049f435-8185-4690-9568-14d9b339013a\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") "
Sep 09 06:03:16.303031 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.302964    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7049f435-8185-4690-9568-14d9b339013a-maas-api-tls\") pod \"7049f435-8185-4690-9568-14d9b339013a\" (UID: \"7049f435-8185-4690-9568-14d9b339013a\") "
Sep 09 06:03:16.303186 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.303156    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7049f435-8185-4690-9568-14d9b339013a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7049f435-8185-4690-9568-14d9b339013a" (UID: "7049f435-8185-4690-9568-14d9b339013a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:03:16.303307 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.303289    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7049f435-8185-4690-9568-14d9b339013a-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:16.305164 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.305127    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7049f435-8185-4690-9568-14d9b339013a-kube-api-access-bh2rb" (OuterVolumeSpecName: "kube-api-access-bh2rb") pod "7049f435-8185-4690-9568-14d9b339013a" (UID: "7049f435-8185-4690-9568-14d9b339013a"). InnerVolumeSpecName "kube-api-access-bh2rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:03:16.305293 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.305255    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7049f435-8185-4690-9568-14d9b339013a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7049f435-8185-4690-9568-14d9b339013a" (UID: "7049f435-8185-4690-9568-14d9b339013a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:03:16.305519 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.305470    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7049f435-8185-4690-9568-14d9b339013a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7049f435-8185-4690-9568-14d9b339013a" (UID: "7049f435-8185-4690-9568-14d9b339013a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:03:16.312532 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.312503    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"]
Sep 09 06:03:16.312823 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.312737    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj" podUID="76716474-bb47-4751-81ac-b09ff1915b00" containerName="istio-proxy" containerID="cri-o://fb185b4619d32782d6896544019b5ccf57ed2fcdda81da498a107f3639ab2a3a" gracePeriod=30
Sep 09 06:03:16.407840 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.407803    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7049f435-8185-4690-9568-14d9b339013a-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:16.408033 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.407871    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7049f435-8185-4690-9568-14d9b339013a-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:16.408033 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.407891    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh2rb\" (UniqueName: \"kubernetes.io/projected/7049f435-8185-4690-9568-14d9b339013a-kube-api-access-bh2rb\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:16.904670 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.904635    2564 generic.go:356] "Generic (PLEG): container finished" podID="7049f435-8185-4690-9568-14d9b339013a" containerID="a0c81a1c8f727782d51b0613a01604c698f72468344deeb1107cd62c0329f916" exitCode=0
Sep 09 06:03:16.904873 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.904682    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"
Sep 09 06:03:16.904873 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.904722    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475" event={"ID":"7049f435-8185-4690-9568-14d9b339013a","Type":"ContainerDied","Data":"a0c81a1c8f727782d51b0613a01604c698f72468344deeb1107cd62c0329f916"}
Sep 09 06:03:16.904873 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.904777    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475" event={"ID":"7049f435-8185-4690-9568-14d9b339013a","Type":"ContainerDied","Data":"2ac3826a2746cfa7790e9ee845c7734650d78a21b6e8c164b020f82ec6de5f4f"}
Sep 09 06:03:16.904873 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.904806    2564 scope.go:117] "RemoveContainer" containerID="a0c81a1c8f727782d51b0613a01604c698f72468344deeb1107cd62c0329f916"
Sep 09 06:03:16.915363 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.915346    2564 scope.go:117] "RemoveContainer" containerID="a0c81a1c8f727782d51b0613a01604c698f72468344deeb1107cd62c0329f916"
Sep 09 06:03:16.915744 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:03:16.915723    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c81a1c8f727782d51b0613a01604c698f72468344deeb1107cd62c0329f916\": container with ID starting with a0c81a1c8f727782d51b0613a01604c698f72468344deeb1107cd62c0329f916 not found: ID does not exist" containerID="a0c81a1c8f727782d51b0613a01604c698f72468344deeb1107cd62c0329f916"
Sep 09 06:03:16.915810 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.915753    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c81a1c8f727782d51b0613a01604c698f72468344deeb1107cd62c0329f916"} err="failed to get container status \"a0c81a1c8f727782d51b0613a01604c698f72468344deeb1107cd62c0329f916\": rpc error: code = NotFound desc = could not find container \"a0c81a1c8f727782d51b0613a01604c698f72468344deeb1107cd62c0329f916\": container with ID starting with a0c81a1c8f727782d51b0613a01604c698f72468344deeb1107cd62c0329f916 not found: ID does not exist"
Sep 09 06:03:16.926217 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.926193    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"]
Sep 09 06:03:16.930930 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:16.930909    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e7b233dd-7475b7f894-mf475"]
Sep 09 06:03:17.303841 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:17.303746    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7049f435-8185-4690-9568-14d9b339013a" path="/var/lib/kubelet/pods/7049f435-8185-4690-9568-14d9b339013a/volumes"
Sep 09 06:03:17.304355 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:17.304262    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="782df8a5-b0b1-4beb-926f-ad47b662efec" path="/var/lib/kubelet/pods/782df8a5-b0b1-4beb-926f-ad47b662efec/volumes"
Sep 09 06:03:21.095908 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.095874    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"]
Sep 09 06:03:21.096330 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.096317    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7049f435-8185-4690-9568-14d9b339013a" containerName="maas-api"
Sep 09 06:03:21.096330 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.096331    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7049f435-8185-4690-9568-14d9b339013a" containerName="maas-api"
Sep 09 06:03:21.096405 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.096357    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="782df8a5-b0b1-4beb-926f-ad47b662efec" containerName="revoke-keys"
Sep 09 06:03:21.096405 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.096363    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="782df8a5-b0b1-4beb-926f-ad47b662efec" containerName="revoke-keys"
Sep 09 06:03:21.096471 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.096433    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7049f435-8185-4690-9568-14d9b339013a" containerName="maas-api"
Sep 09 06:03:21.096471 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.096442    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="782df8a5-b0b1-4beb-926f-ad47b662efec" containerName="revoke-keys"
Sep 09 06:03:21.106703 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.104684    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:21.110669 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.110622    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8b70f171-serving-cert\""
Sep 09 06:03:21.111990 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.111955    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"]
Sep 09 06:03:21.258114 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.258069    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f482d1-d171-48c5-b43f-f56b2216f950-openshift-service-ca\") pod \"maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:21.258306 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.258161    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcq5w\" (UniqueName: \"kubernetes.io/projected/b7f482d1-d171-48c5-b43f-f56b2216f950-kube-api-access-gcq5w\") pod \"maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:21.258306 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.258212    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-maas-api-tls\") pod \"maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:21.258306 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.258268    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-metrics-certs\") pod \"maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:21.359830 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.359760    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-maas-api-tls\") pod \"maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:21.359830 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.359802    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-metrics-certs\") pod \"maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:21.360037 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.359854    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f482d1-d171-48c5-b43f-f56b2216f950-openshift-service-ca\") pod \"maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:21.360037 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.359898    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcq5w\" (UniqueName: \"kubernetes.io/projected/b7f482d1-d171-48c5-b43f-f56b2216f950-kube-api-access-gcq5w\") pod \"maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:21.360037 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:03:21.359918    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-serving-cert: secret "maas-api-e2e-ait-8b70f171-serving-cert" not found
Sep 09 06:03:21.360037 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:03:21.359993    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-maas-api-tls podName:b7f482d1-d171-48c5-b43f-f56b2216f950 nodeName:}" failed. No retries permitted until 2026-09-09 06:03:21.85997137 +0000 UTC m=+829.076301232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-maas-api-tls") pod "maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47" (UID: "b7f482d1-d171-48c5-b43f-f56b2216f950") : secret "maas-api-e2e-ait-8b70f171-serving-cert" not found
Sep 09 06:03:21.360504 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.360471    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f482d1-d171-48c5-b43f-f56b2216f950-openshift-service-ca\") pod \"maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:21.362149 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.362133    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-metrics-certs\") pod \"maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:21.369524 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.369485    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcq5w\" (UniqueName: \"kubernetes.io/projected/b7f482d1-d171-48c5-b43f-f56b2216f950-kube-api-access-gcq5w\") pod \"maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:21.581273 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.581243    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:03:21.664278 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.664187    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-workload-socket\") pod \"76716474-bb47-4751-81ac-b09ff1915b00\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") "
Sep 09 06:03:21.664278 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.664240    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76716474-bb47-4751-81ac-b09ff1915b00-istio-token\") pod \"76716474-bb47-4751-81ac-b09ff1915b00\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") "
Sep 09 06:03:21.664530 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.664285    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-istio-data\") pod \"76716474-bb47-4751-81ac-b09ff1915b00\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") "
Sep 09 06:03:21.664530 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.664308    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76716474-bb47-4751-81ac-b09ff1915b00-istiod-ca-cert\") pod \"76716474-bb47-4751-81ac-b09ff1915b00\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") "
Sep 09 06:03:21.664530 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.664336    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-credential-socket\") pod \"76716474-bb47-4751-81ac-b09ff1915b00\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") "
Sep 09 06:03:21.664530 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.664364    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76716474-bb47-4751-81ac-b09ff1915b00-istio-podinfo\") pod \"76716474-bb47-4751-81ac-b09ff1915b00\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") "
Sep 09 06:03:21.664530 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.664389    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-workload-certs\") pod \"76716474-bb47-4751-81ac-b09ff1915b00\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") "
Sep 09 06:03:21.664530 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.664436    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4bqh\" (UniqueName: \"kubernetes.io/projected/76716474-bb47-4751-81ac-b09ff1915b00-kube-api-access-n4bqh\") pod \"76716474-bb47-4751-81ac-b09ff1915b00\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") "
Sep 09 06:03:21.664530 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.664483    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "76716474-bb47-4751-81ac-b09ff1915b00" (UID: "76716474-bb47-4751-81ac-b09ff1915b00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:03:21.664891 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.664556    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-istio-envoy\") pod \"76716474-bb47-4751-81ac-b09ff1915b00\" (UID: \"76716474-bb47-4751-81ac-b09ff1915b00\") "
Sep 09 06:03:21.664891 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.664681    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "76716474-bb47-4751-81ac-b09ff1915b00" (UID: "76716474-bb47-4751-81ac-b09ff1915b00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:03:21.664891 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.664759    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "76716474-bb47-4751-81ac-b09ff1915b00" (UID: "76716474-bb47-4751-81ac-b09ff1915b00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:03:21.664891 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.664806    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76716474-bb47-4751-81ac-b09ff1915b00-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "76716474-bb47-4751-81ac-b09ff1915b00" (UID: "76716474-bb47-4751-81ac-b09ff1915b00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:03:21.665097 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.665016    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-istio-data" (OuterVolumeSpecName: "istio-data") pod "76716474-bb47-4751-81ac-b09ff1915b00" (UID: "76716474-bb47-4751-81ac-b09ff1915b00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:03:21.665097 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.665067    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76716474-bb47-4751-81ac-b09ff1915b00-istiod-ca-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:21.665201 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.665094    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-credential-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:21.665201 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.665110    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-workload-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:21.665201 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.665124    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-workload-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:21.666897 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.666862    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "76716474-bb47-4751-81ac-b09ff1915b00" (UID: "76716474-bb47-4751-81ac-b09ff1915b00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:03:21.666897 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.666878    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76716474-bb47-4751-81ac-b09ff1915b00-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "76716474-bb47-4751-81ac-b09ff1915b00" (UID: "76716474-bb47-4751-81ac-b09ff1915b00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 06:03:21.667079 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.666902    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76716474-bb47-4751-81ac-b09ff1915b00-istio-token" (OuterVolumeSpecName: "istio-token") pod "76716474-bb47-4751-81ac-b09ff1915b00" (UID: "76716474-bb47-4751-81ac-b09ff1915b00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:03:21.667079 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.667041    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76716474-bb47-4751-81ac-b09ff1915b00-kube-api-access-n4bqh" (OuterVolumeSpecName: "kube-api-access-n4bqh") pod "76716474-bb47-4751-81ac-b09ff1915b00" (UID: "76716474-bb47-4751-81ac-b09ff1915b00"). InnerVolumeSpecName "kube-api-access-n4bqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:03:21.766270 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.766236    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76716474-bb47-4751-81ac-b09ff1915b00-istio-podinfo\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:21.766270 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.766269    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4bqh\" (UniqueName: \"kubernetes.io/projected/76716474-bb47-4751-81ac-b09ff1915b00-kube-api-access-n4bqh\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:21.766476 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.766283    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-istio-envoy\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:21.766476 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.766294    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76716474-bb47-4751-81ac-b09ff1915b00-istio-token\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:21.766476 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.766305    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76716474-bb47-4751-81ac-b09ff1915b00-istio-data\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:21.867680 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.867641    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-maas-api-tls\") pod \"maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:21.867837 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:03:21.867799    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-serving-cert: secret "maas-api-e2e-ait-8b70f171-serving-cert" not found
Sep 09 06:03:21.867906 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:03:21.867888    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-maas-api-tls podName:b7f482d1-d171-48c5-b43f-f56b2216f950 nodeName:}" failed. No retries permitted until 2026-09-09 06:03:22.867872274 +0000 UTC m=+830.084202136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-maas-api-tls") pod "maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47" (UID: "b7f482d1-d171-48c5-b43f-f56b2216f950") : secret "maas-api-e2e-ait-8b70f171-serving-cert" not found
Sep 09 06:03:21.937220 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.937130    2564 generic.go:356] "Generic (PLEG): container finished" podID="76716474-bb47-4751-81ac-b09ff1915b00" containerID="fb185b4619d32782d6896544019b5ccf57ed2fcdda81da498a107f3639ab2a3a" exitCode=0
Sep 09 06:03:21.937220 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.937175    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj" event={"ID":"76716474-bb47-4751-81ac-b09ff1915b00","Type":"ContainerDied","Data":"fb185b4619d32782d6896544019b5ccf57ed2fcdda81da498a107f3639ab2a3a"}
Sep 09 06:03:21.937220 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.937180    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"
Sep 09 06:03:21.937439 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.937221    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj" event={"ID":"76716474-bb47-4751-81ac-b09ff1915b00","Type":"ContainerDied","Data":"94419cc28e124db6a6f32d85dd3a8a42b4608c5ac5208b69e8d23b0aeb5d7688"}
Sep 09 06:03:21.937439 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.937248    2564 scope.go:117] "RemoveContainer" containerID="fb185b4619d32782d6896544019b5ccf57ed2fcdda81da498a107f3639ab2a3a"
Sep 09 06:03:21.948866 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.948838    2564 scope.go:117] "RemoveContainer" containerID="48a679ec70a61e682138249c9a26532166b0382b466e0cb397c554a8555b977f"
Sep 09 06:03:21.957952 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.957931    2564 scope.go:117] "RemoveContainer" containerID="fb185b4619d32782d6896544019b5ccf57ed2fcdda81da498a107f3639ab2a3a"
Sep 09 06:03:21.958245 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:03:21.958218    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb185b4619d32782d6896544019b5ccf57ed2fcdda81da498a107f3639ab2a3a\": container with ID starting with fb185b4619d32782d6896544019b5ccf57ed2fcdda81da498a107f3639ab2a3a not found: ID does not exist" containerID="fb185b4619d32782d6896544019b5ccf57ed2fcdda81da498a107f3639ab2a3a"
Sep 09 06:03:21.958320 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.958251    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb185b4619d32782d6896544019b5ccf57ed2fcdda81da498a107f3639ab2a3a"} err="failed to get container status \"fb185b4619d32782d6896544019b5ccf57ed2fcdda81da498a107f3639ab2a3a\": rpc error: code = NotFound desc = could not find container \"fb185b4619d32782d6896544019b5ccf57ed2fcdda81da498a107f3639ab2a3a\": container with ID starting with fb185b4619d32782d6896544019b5ccf57ed2fcdda81da498a107f3639ab2a3a not found: ID does not exist"
Sep 09 06:03:21.958320 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.958274    2564 scope.go:117] "RemoveContainer" containerID="48a679ec70a61e682138249c9a26532166b0382b466e0cb397c554a8555b977f"
Sep 09 06:03:21.958610 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:03:21.958574    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a679ec70a61e682138249c9a26532166b0382b466e0cb397c554a8555b977f\": container with ID starting with 48a679ec70a61e682138249c9a26532166b0382b466e0cb397c554a8555b977f not found: ID does not exist" containerID="48a679ec70a61e682138249c9a26532166b0382b466e0cb397c554a8555b977f"
Sep 09 06:03:21.958706 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.958611    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a679ec70a61e682138249c9a26532166b0382b466e0cb397c554a8555b977f"} err="failed to get container status \"48a679ec70a61e682138249c9a26532166b0382b466e0cb397c554a8555b977f\": rpc error: code = NotFound desc = could not find container \"48a679ec70a61e682138249c9a26532166b0382b466e0cb397c554a8555b977f\": container with ID starting with 48a679ec70a61e682138249c9a26532166b0382b466e0cb397c554a8555b977f not found: ID does not exist"
Sep 09 06:03:21.960001 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.959976    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"]
Sep 09 06:03:21.965785 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:21.965766    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e7b233dd-gw-openshift-default-7957cd89f6-hwcmj"]
Sep 09 06:03:22.876525 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:22.876462    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-maas-api-tls\") pod \"maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:22.878930 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:22.878896    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-maas-api-tls\") pod \"maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:22.921466 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:22.921436    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:23.210951 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:23.210927    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"]
Sep 09 06:03:23.213126 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:03:23.213099    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f482d1_d171_48c5_b43f_f56b2216f950.slice/crio-0aee99b91de41089bc09a0dbdc655beff1412730cdc93f469ecedac8f2818f7a WatchSource:0}: Error finding container 0aee99b91de41089bc09a0dbdc655beff1412730cdc93f469ecedac8f2818f7a: Status 404 returned error can't find the container with id 0aee99b91de41089bc09a0dbdc655beff1412730cdc93f469ecedac8f2818f7a
Sep 09 06:03:23.303414 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:23.303381    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76716474-bb47-4751-81ac-b09ff1915b00" path="/var/lib/kubelet/pods/76716474-bb47-4751-81ac-b09ff1915b00/volumes"
Sep 09 06:03:23.966343 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:23.966246    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47" event={"ID":"b7f482d1-d171-48c5-b43f-f56b2216f950","Type":"ContainerStarted","Data":"e4487f4148b06c79826f3843778e2525da77964ed73a7b591cf64a7fe0137a34"}
Sep 09 06:03:23.966731 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:23.966379    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:23.966731 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:23.966425    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47" event={"ID":"b7f482d1-d171-48c5-b43f-f56b2216f950","Type":"ContainerStarted","Data":"0aee99b91de41089bc09a0dbdc655beff1412730cdc93f469ecedac8f2818f7a"}
Sep 09 06:03:23.982028 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:23.981984    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47" podStartSLOduration=2.6233251170000003 podStartE2EDuration="2.981970965s" totalImagesPullingTime="358.645848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:03:23.981219994 +0000 UTC m=+831.197549878" watchObservedRunningTime="2026-09-09 06:03:23.981970965 +0000 UTC m=+831.198300848"
Sep 09 06:03:29.977652 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:29.977617    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:42.127735 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.127699    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"]
Sep 09 06:03:42.128177 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.128164    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76716474-bb47-4751-81ac-b09ff1915b00" containerName="istio-proxy"
Sep 09 06:03:42.128222 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.128179    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="76716474-bb47-4751-81ac-b09ff1915b00" containerName="istio-proxy"
Sep 09 06:03:42.128222 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.128191    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76716474-bb47-4751-81ac-b09ff1915b00" containerName="istio-proxy"
Sep 09 06:03:42.128222 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.128196    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="76716474-bb47-4751-81ac-b09ff1915b00" containerName="istio-proxy"
Sep 09 06:03:42.128358 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.128290    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="76716474-bb47-4751-81ac-b09ff1915b00" containerName="istio-proxy"
Sep 09 06:03:42.128358 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.128300    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="76716474-bb47-4751-81ac-b09ff1915b00" containerName="istio-proxy"
Sep 09 06:03:42.133296 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.133277    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.136650 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.136616    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ecd13e\"/\"openshift-service-ca.crt\""
Sep 09 06:03:42.136650 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.136637    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ecd13e\"/\"test-model-ecd13e-kserve-self-signed-certs\""
Sep 09 06:03:42.136847 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.136637    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ecd13e\"/\"default-dockercfg-5slxk\""
Sep 09 06:03:42.136847 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.136619    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ecd13e\"/\"kube-root-ca.crt\""
Sep 09 06:03:42.141982 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.141944    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"]
Sep 09 06:03:42.163603 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.163574    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da978ab-db7e-44ea-893e-cd60c07bae07-tls-certs\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.163790 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.163614    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-tmp-dir\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.163790 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.163643    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-dshm\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.163790 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.163677    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbrg\" (UniqueName: \"kubernetes.io/projected/0da978ab-db7e-44ea-893e-cd60c07bae07-kube-api-access-kgbrg\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.163947 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.163875    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-model-cache\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.163999 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.163967    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-home\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.264927 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.264887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbrg\" (UniqueName: \"kubernetes.io/projected/0da978ab-db7e-44ea-893e-cd60c07bae07-kube-api-access-kgbrg\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.265105 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.264981    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-model-cache\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.265105 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.265008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-home\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.265105 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.265032    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da978ab-db7e-44ea-893e-cd60c07bae07-tls-certs\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.265105 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.265049    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-tmp-dir\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.265105 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.265067    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-dshm\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.265449 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.265425    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-tmp-dir\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.265449 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.265440    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-model-cache\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.265581 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.265511    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-home\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.267367 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.267344    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-dshm\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.267687 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.267668    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da978ab-db7e-44ea-893e-cd60c07bae07-tls-certs\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.272962 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.272940    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbrg\" (UniqueName: \"kubernetes.io/projected/0da978ab-db7e-44ea-893e-cd60c07bae07-kube-api-access-kgbrg\") pod \"test-model-ecd13e-kserve-6c64c5b887-px62z\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") " pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.444282 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.444196    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:42.720934 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:42.720905    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"]
Sep 09 06:03:42.721884 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:03:42.721859    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0da978ab_db7e_44ea_893e_cd60c07bae07.slice/crio-1dad9c80cb6689700eaa3f8118636645e6d47b926dbb2dd7fe99fd160efadf80 WatchSource:0}: Error finding container 1dad9c80cb6689700eaa3f8118636645e6d47b926dbb2dd7fe99fd160efadf80: Status 404 returned error can't find the container with id 1dad9c80cb6689700eaa3f8118636645e6d47b926dbb2dd7fe99fd160efadf80
Sep 09 06:03:43.086234 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:43.086191    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z" event={"ID":"0da978ab-db7e-44ea-893e-cd60c07bae07","Type":"ContainerStarted","Data":"413db7d5fa1756bfe36d375d27b2c80d9d0568eaaaf688e1ebb7b5d54bfc938f"}
Sep 09 06:03:43.086430 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:43.086247    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z" event={"ID":"0da978ab-db7e-44ea-893e-cd60c07bae07","Type":"ContainerStarted","Data":"1dad9c80cb6689700eaa3f8118636645e6d47b926dbb2dd7fe99fd160efadf80"}
Sep 09 06:03:43.104461 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:43.104405    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z" podStartSLOduration=1.104384534 podStartE2EDuration="1.104384534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:03:43.101275652 +0000 UTC m=+850.317605537" watchObservedRunningTime="2026-09-09 06:03:43.104384534 +0000 UTC m=+850.320714434"
Sep 09 06:03:44.817105 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:44.817069    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc"]
Sep 09 06:03:44.820772 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:44.820752    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc"
Sep 09 06:03:44.823602 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:44.823580    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:03:44.830148 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:44.830104    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc"]
Sep 09 06:03:44.893849 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:44.893820    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7a357a-h6jfc\" (UID: \"6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc"
Sep 09 06:03:44.995311 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:44.995275    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7a357a-h6jfc\" (UID: \"6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc"
Sep 09 06:03:44.995895 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:44.995877    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7a357a-h6jfc\" (UID: \"6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc"
Sep 09 06:03:45.135216 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:45.135183    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc"
Sep 09 06:03:45.411995 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:45.411966    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc"]
Sep 09 06:03:45.412913 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:03:45.412889    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6add7f7f_07f4_4a6d_bbc6_dbd1e5e2cfeb.slice/crio-3035576a776e105fb75938d6eb38e85c5dc1a66369acfce088a51b0f5587a20e WatchSource:0}: Error finding container 3035576a776e105fb75938d6eb38e85c5dc1a66369acfce088a51b0f5587a20e: Status 404 returned error can't find the container with id 3035576a776e105fb75938d6eb38e85c5dc1a66369acfce088a51b0f5587a20e
Sep 09 06:03:46.106362 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:46.106327    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc" event={"ID":"6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb","Type":"ContainerStarted","Data":"7e2c239c9efb0518dfeaf97dc5a06f2d66d01c646dc7102d29e9846d46771138"}
Sep 09 06:03:46.106758 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:46.106381    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc" event={"ID":"6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb","Type":"ContainerStarted","Data":"3035576a776e105fb75938d6eb38e85c5dc1a66369acfce088a51b0f5587a20e"}
Sep 09 06:03:46.121759 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:46.121710    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc" podStartSLOduration=2.121694733 podStartE2EDuration="2.121694733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:03:46.119282237 +0000 UTC m=+853.335612121" watchObservedRunningTime="2026-09-09 06:03:46.121694733 +0000 UTC m=+853.338024617"
Sep 09 06:03:47.113806 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:47.113772    2564 generic.go:356] "Generic (PLEG): container finished" podID="6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb" containerID="7e2c239c9efb0518dfeaf97dc5a06f2d66d01c646dc7102d29e9846d46771138" exitCode=0
Sep 09 06:03:47.114187 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:47.113855    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc" event={"ID":"6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb","Type":"ContainerDied","Data":"7e2c239c9efb0518dfeaf97dc5a06f2d66d01c646dc7102d29e9846d46771138"}
Sep 09 06:03:48.262315 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:48.262290    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc"
Sep 09 06:03:48.327519 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:48.327461    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb-maas-api-service-ca\") pod \"6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb\" (UID: \"6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb\") "
Sep 09 06:03:48.327843 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:48.327817    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb" (UID: "6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:03:48.429347 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:48.429240    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb-maas-api-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:49.128386 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:49.128360    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc"
Sep 09 06:03:49.128578 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:49.128396    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc" event={"ID":"6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb","Type":"ContainerDied","Data":"3035576a776e105fb75938d6eb38e85c5dc1a66369acfce088a51b0f5587a20e"}
Sep 09 06:03:49.128578 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:49.128446    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3035576a776e105fb75938d6eb38e85c5dc1a66369acfce088a51b0f5587a20e"
Sep 09 06:03:51.723186 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:51.723151    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"]
Sep 09 06:03:51.723630 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:51.723402    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47" podUID="b7f482d1-d171-48c5-b43f-f56b2216f950" containerName="maas-api" containerID="cri-o://e4487f4148b06c79826f3843778e2525da77964ed73a7b591cf64a7fe0137a34" gracePeriod=30
Sep 09 06:03:51.975112 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:51.975052    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:52.063644 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.063610    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-maas-api-tls\") pod \"b7f482d1-d171-48c5-b43f-f56b2216f950\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") "
Sep 09 06:03:52.063836 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.063653    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-metrics-certs\") pod \"b7f482d1-d171-48c5-b43f-f56b2216f950\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") "
Sep 09 06:03:52.063836 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.063684    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f482d1-d171-48c5-b43f-f56b2216f950-openshift-service-ca\") pod \"b7f482d1-d171-48c5-b43f-f56b2216f950\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") "
Sep 09 06:03:52.063836 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.063756    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcq5w\" (UniqueName: \"kubernetes.io/projected/b7f482d1-d171-48c5-b43f-f56b2216f950-kube-api-access-gcq5w\") pod \"b7f482d1-d171-48c5-b43f-f56b2216f950\" (UID: \"b7f482d1-d171-48c5-b43f-f56b2216f950\") "
Sep 09 06:03:52.064045 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.064002    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f482d1-d171-48c5-b43f-f56b2216f950-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b7f482d1-d171-48c5-b43f-f56b2216f950" (UID: "b7f482d1-d171-48c5-b43f-f56b2216f950"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:03:52.064155 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.064132    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f482d1-d171-48c5-b43f-f56b2216f950-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:52.065858 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.065822    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b7f482d1-d171-48c5-b43f-f56b2216f950" (UID: "b7f482d1-d171-48c5-b43f-f56b2216f950"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:03:52.065961 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.065860    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b7f482d1-d171-48c5-b43f-f56b2216f950" (UID: "b7f482d1-d171-48c5-b43f-f56b2216f950"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:03:52.065961 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.065881    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f482d1-d171-48c5-b43f-f56b2216f950-kube-api-access-gcq5w" (OuterVolumeSpecName: "kube-api-access-gcq5w") pod "b7f482d1-d171-48c5-b43f-f56b2216f950" (UID: "b7f482d1-d171-48c5-b43f-f56b2216f950"). InnerVolumeSpecName "kube-api-access-gcq5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:03:52.152950 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.152918    2564 generic.go:356] "Generic (PLEG): container finished" podID="b7f482d1-d171-48c5-b43f-f56b2216f950" containerID="e4487f4148b06c79826f3843778e2525da77964ed73a7b591cf64a7fe0137a34" exitCode=0
Sep 09 06:03:52.153126 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.152963    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"
Sep 09 06:03:52.153126 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.153006    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47" event={"ID":"b7f482d1-d171-48c5-b43f-f56b2216f950","Type":"ContainerDied","Data":"e4487f4148b06c79826f3843778e2525da77964ed73a7b591cf64a7fe0137a34"}
Sep 09 06:03:52.153126 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.153054    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47" event={"ID":"b7f482d1-d171-48c5-b43f-f56b2216f950","Type":"ContainerDied","Data":"0aee99b91de41089bc09a0dbdc655beff1412730cdc93f469ecedac8f2818f7a"}
Sep 09 06:03:52.153126 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.153078    2564 scope.go:117] "RemoveContainer" containerID="e4487f4148b06c79826f3843778e2525da77964ed73a7b591cf64a7fe0137a34"
Sep 09 06:03:52.163588 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.163549    2564 scope.go:117] "RemoveContainer" containerID="e4487f4148b06c79826f3843778e2525da77964ed73a7b591cf64a7fe0137a34"
Sep 09 06:03:52.163913 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:03:52.163888    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4487f4148b06c79826f3843778e2525da77964ed73a7b591cf64a7fe0137a34\": container with ID starting with e4487f4148b06c79826f3843778e2525da77964ed73a7b591cf64a7fe0137a34 not found: ID does not exist" containerID="e4487f4148b06c79826f3843778e2525da77964ed73a7b591cf64a7fe0137a34"
Sep 09 06:03:52.164013 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.163923    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4487f4148b06c79826f3843778e2525da77964ed73a7b591cf64a7fe0137a34"} err="failed to get container status \"e4487f4148b06c79826f3843778e2525da77964ed73a7b591cf64a7fe0137a34\": rpc error: code = NotFound desc = could not find container \"e4487f4148b06c79826f3843778e2525da77964ed73a7b591cf64a7fe0137a34\": container with ID starting with e4487f4148b06c79826f3843778e2525da77964ed73a7b591cf64a7fe0137a34 not found: ID does not exist"
Sep 09 06:03:52.164677 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.164656    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcq5w\" (UniqueName: \"kubernetes.io/projected/b7f482d1-d171-48c5-b43f-f56b2216f950-kube-api-access-gcq5w\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:52.164756 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.164678    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:52.164756 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.164692    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7f482d1-d171-48c5-b43f-f56b2216f950-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:52.176550 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.176415    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"]
Sep 09 06:03:52.179814 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.179793    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b70f171-5c9dbffcf5-twj47"]
Sep 09 06:03:52.444971 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.444931    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:52.445246 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.445223    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:52.457950 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:52.457926    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:53.176433 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.176405    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:03:53.303483 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.303452    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7f482d1-d171-48c5-b43f-f56b2216f950" path="/var/lib/kubelet/pods/b7f482d1-d171-48c5-b43f-f56b2216f950/volumes"
Sep 09 06:03:53.768098 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.768069    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"]
Sep 09 06:03:53.768570 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.768553    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb" containerName="revoke-keys"
Sep 09 06:03:53.768635 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.768574    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb" containerName="revoke-keys"
Sep 09 06:03:53.768635 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.768595    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7f482d1-d171-48c5-b43f-f56b2216f950" containerName="maas-api"
Sep 09 06:03:53.768635 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.768603    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7f482d1-d171-48c5-b43f-f56b2216f950" containerName="maas-api"
Sep 09 06:03:53.768744 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.768690    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7f482d1-d171-48c5-b43f-f56b2216f950" containerName="maas-api"
Sep 09 06:03:53.768744 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.768701    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb" containerName="revoke-keys"
Sep 09 06:03:53.773094 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.773074    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.775267 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.775247    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d1d9fd16-openshift-default-dockercfg-m5pdd\""
Sep 09 06:03:53.780444 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.780422    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"]
Sep 09 06:03:53.881402 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.881360    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-workload-socket\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.881402 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.881398    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-workload-certs\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.881643 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.881438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0de45e-3b05-46a0-a50a-c5c62741425a-istiod-ca-cert\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.881643 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.881474    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-data\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.881643 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.881536    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltdrp\" (UniqueName: \"kubernetes.io/projected/9a0de45e-3b05-46a0-a50a-c5c62741425a-kube-api-access-ltdrp\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.881643 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.881604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-token\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.881803 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.881648    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-credential-socket\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.881803 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.881691    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-envoy\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.881803 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.881716    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-podinfo\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.982955 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.982917    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-envoy\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.983123 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.982975    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-podinfo\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.983123 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.983046    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-workload-socket\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.983123 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.983073    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-workload-certs\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.983123 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.983113    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0de45e-3b05-46a0-a50a-c5c62741425a-istiod-ca-cert\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.983357 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.983150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-data\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.983357 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.983180    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdrp\" (UniqueName: \"kubernetes.io/projected/9a0de45e-3b05-46a0-a50a-c5c62741425a-kube-api-access-ltdrp\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.983357 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.983227    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-token\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.983357 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.983268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-credential-socket\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.983606 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.983566    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-workload-certs\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.983664 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.983618    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-data\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.983722 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.983680    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-credential-socket\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.983782 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.983759    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-workload-socket\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.984011 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.983983    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0de45e-3b05-46a0-a50a-c5c62741425a-istiod-ca-cert\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.985417 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.985391    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-podinfo\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.985570 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.985554    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-envoy\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.992233 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.992211    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdrp\" (UniqueName: \"kubernetes.io/projected/9a0de45e-3b05-46a0-a50a-c5c62741425a-kube-api-access-ltdrp\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:53.999772 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:53.999751    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-token\") pod \"e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") " pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:54.084571 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:54.084538    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:54.363898 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:54.363872    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"]
Sep 09 06:03:54.365113 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:03:54.365092    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0de45e_3b05_46a0_a50a_c5c62741425a.slice/crio-3e6507bf5bee96ed73d15949e0af123eb2e39a835d904ae4c74f369e6420a19b WatchSource:0}: Error finding container 3e6507bf5bee96ed73d15949e0af123eb2e39a835d904ae4c74f369e6420a19b: Status 404 returned error can't find the container with id 3e6507bf5bee96ed73d15949e0af123eb2e39a835d904ae4c74f369e6420a19b
Sep 09 06:03:54.367259 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:54.367228    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:03:54.367332 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:54.367287    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:03:54.367332 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:54.367315    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:03:55.177120 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:55.177074    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25" event={"ID":"9a0de45e-3b05-46a0-a50a-c5c62741425a","Type":"ContainerStarted","Data":"a3c0da4467a192963c2bf1e12f551d883bc960e62474911f4219c81ffb988be3"}
Sep 09 06:03:55.177295 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:55.177173    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25" event={"ID":"9a0de45e-3b05-46a0-a50a-c5c62741425a","Type":"ContainerStarted","Data":"3e6507bf5bee96ed73d15949e0af123eb2e39a835d904ae4c74f369e6420a19b"}
Sep 09 06:03:55.197356 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:55.197237    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25" podStartSLOduration=2.197216091 podStartE2EDuration="2.197216091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:03:55.195638347 +0000 UTC m=+862.411968234" watchObservedRunningTime="2026-09-09 06:03:55.197216091 +0000 UTC m=+862.413545976"
Sep 09 06:03:55.716033 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:55.715999    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"]
Sep 09 06:03:55.716467 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:55.716213    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f" podUID="910a5724-7ee3-4490-b58b-cc590f14ecf3" containerName="payload-pre-processing" containerID="cri-o://355a64036b6ff11841cdeced272ea17dcedf5d7001f80e34eaa68089e3d7ed88" gracePeriod=30
Sep 09 06:03:55.862464 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:55.862421    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc"]
Sep 09 06:03:55.865087 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:55.865061    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7a357a-h6jfc"]
Sep 09 06:03:55.984121 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:55.984094    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"
Sep 09 06:03:56.085519 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.085467    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:56.091098 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.091069    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:56.103539 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.103486    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910a5724-7ee3-4490-b58b-cc590f14ecf3-plugins-config-volume\") pod \"910a5724-7ee3-4490-b58b-cc590f14ecf3\" (UID: \"910a5724-7ee3-4490-b58b-cc590f14ecf3\") "
Sep 09 06:03:56.103684 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.103595    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqhj4\" (UniqueName: \"kubernetes.io/projected/910a5724-7ee3-4490-b58b-cc590f14ecf3-kube-api-access-tqhj4\") pod \"910a5724-7ee3-4490-b58b-cc590f14ecf3\" (UID: \"910a5724-7ee3-4490-b58b-cc590f14ecf3\") "
Sep 09 06:03:56.103833 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.103810    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/910a5724-7ee3-4490-b58b-cc590f14ecf3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "910a5724-7ee3-4490-b58b-cc590f14ecf3" (UID: "910a5724-7ee3-4490-b58b-cc590f14ecf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:03:56.105849 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.105827    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/910a5724-7ee3-4490-b58b-cc590f14ecf3-kube-api-access-tqhj4" (OuterVolumeSpecName: "kube-api-access-tqhj4") pod "910a5724-7ee3-4490-b58b-cc590f14ecf3" (UID: "910a5724-7ee3-4490-b58b-cc590f14ecf3"). InnerVolumeSpecName "kube-api-access-tqhj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:03:56.184301 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.184267    2564 generic.go:356] "Generic (PLEG): container finished" podID="910a5724-7ee3-4490-b58b-cc590f14ecf3" containerID="355a64036b6ff11841cdeced272ea17dcedf5d7001f80e34eaa68089e3d7ed88" exitCode=0
Sep 09 06:03:56.184476 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.184315    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"
Sep 09 06:03:56.184476 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.184347    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f" event={"ID":"910a5724-7ee3-4490-b58b-cc590f14ecf3","Type":"ContainerDied","Data":"355a64036b6ff11841cdeced272ea17dcedf5d7001f80e34eaa68089e3d7ed88"}
Sep 09 06:03:56.184476 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.184398    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f" event={"ID":"910a5724-7ee3-4490-b58b-cc590f14ecf3","Type":"ContainerDied","Data":"8d7c83a45bc9a2f89cec54d16b2c8872f4e3eee442865c1aa863c6908910a43b"}
Sep 09 06:03:56.184476 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.184426    2564 scope.go:117] "RemoveContainer" containerID="355a64036b6ff11841cdeced272ea17dcedf5d7001f80e34eaa68089e3d7ed88"
Sep 09 06:03:56.184935 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.184912    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:56.186047 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.186028    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:03:56.194393 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.194380    2564 scope.go:117] "RemoveContainer" containerID="355a64036b6ff11841cdeced272ea17dcedf5d7001f80e34eaa68089e3d7ed88"
Sep 09 06:03:56.194670 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:03:56.194654    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355a64036b6ff11841cdeced272ea17dcedf5d7001f80e34eaa68089e3d7ed88\": container with ID starting with 355a64036b6ff11841cdeced272ea17dcedf5d7001f80e34eaa68089e3d7ed88 not found: ID does not exist" containerID="355a64036b6ff11841cdeced272ea17dcedf5d7001f80e34eaa68089e3d7ed88"
Sep 09 06:03:56.194709 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.194678    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355a64036b6ff11841cdeced272ea17dcedf5d7001f80e34eaa68089e3d7ed88"} err="failed to get container status \"355a64036b6ff11841cdeced272ea17dcedf5d7001f80e34eaa68089e3d7ed88\": rpc error: code = NotFound desc = could not find container \"355a64036b6ff11841cdeced272ea17dcedf5d7001f80e34eaa68089e3d7ed88\": container with ID starting with 355a64036b6ff11841cdeced272ea17dcedf5d7001f80e34eaa68089e3d7ed88 not found: ID does not exist"
Sep 09 06:03:56.205482 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.205459    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910a5724-7ee3-4490-b58b-cc590f14ecf3-plugins-config-volume\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:56.205610 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.205488    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqhj4\" (UniqueName: \"kubernetes.io/projected/910a5724-7ee3-4490-b58b-cc590f14ecf3-kube-api-access-tqhj4\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:03:56.224183 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.224099    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"]
Sep 09 06:03:56.229754 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:56.229462    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7a357a-7b44ff74b-6qk4f"]
Sep 09 06:03:57.303384 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.303350    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb" path="/var/lib/kubelet/pods/6add7f7f-07f4-4a6d-bbc6-dbd1e5e2cfeb/volumes"
Sep 09 06:03:57.303799 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.303783    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="910a5724-7ee3-4490-b58b-cc590f14ecf3" path="/var/lib/kubelet/pods/910a5724-7ee3-4490-b58b-cc590f14ecf3/volumes"
Sep 09 06:03:57.740806 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.740770    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"]
Sep 09 06:03:57.741454 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.741438    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="910a5724-7ee3-4490-b58b-cc590f14ecf3" containerName="payload-pre-processing"
Sep 09 06:03:57.741535 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.741459    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="910a5724-7ee3-4490-b58b-cc590f14ecf3" containerName="payload-pre-processing"
Sep 09 06:03:57.741626 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.741611    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="910a5724-7ee3-4490-b58b-cc590f14ecf3" containerName="payload-pre-processing"
Sep 09 06:03:57.747080 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.747059    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:03:57.749756 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.749733    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d1d9fd16-serving-cert\""
Sep 09 06:03:57.753427 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.753400    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"]
Sep 09 06:03:57.822267 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.822234    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-maas-api-tls\") pod \"maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:03:57.822267 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.822267    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-openshift-service-ca\") pod \"maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:03:57.822558 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.822290    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-metrics-certs\") pod \"maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:03:57.822558 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.822404    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgcqw\" (UniqueName: \"kubernetes.io/projected/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-kube-api-access-cgcqw\") pod \"maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:03:57.923673 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.923630    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-maas-api-tls\") pod \"maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:03:57.923673 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.923672    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-openshift-service-ca\") pod \"maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:03:57.923889 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:03:57.923787    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-serving-cert: secret "maas-api-e2e-derive-d1d9fd16-serving-cert" not found
Sep 09 06:03:57.923889 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.923785    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-metrics-certs\") pod \"maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:03:57.923889 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:03:57.923854    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-maas-api-tls podName:34272d09-7edc-4e8e-95f4-a3a1466b6c2a nodeName:}" failed. No retries permitted until 2026-09-09 06:03:58.423831925 +0000 UTC m=+865.640161789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-maas-api-tls") pod "maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc" (UID: "34272d09-7edc-4e8e-95f4-a3a1466b6c2a") : secret "maas-api-e2e-derive-d1d9fd16-serving-cert" not found
Sep 09 06:03:57.924038 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.923921    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgcqw\" (UniqueName: \"kubernetes.io/projected/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-kube-api-access-cgcqw\") pod \"maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:03:57.924288 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.924269    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-openshift-service-ca\") pod \"maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:03:57.926150 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.926131    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-metrics-certs\") pod \"maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:03:57.934480 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:57.934460    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgcqw\" (UniqueName: \"kubernetes.io/projected/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-kube-api-access-cgcqw\") pod \"maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:03:58.430431 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:58.430387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-maas-api-tls\") pod \"maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:03:58.432891 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:58.432868    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-maas-api-tls\") pod \"maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:03:58.662083 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:58.661984    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:03:58.943333 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:58.943302    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"]
Sep 09 06:03:58.945173 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:03:58.945148    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34272d09_7edc_4e8e_95f4_a3a1466b6c2a.slice/crio-93ec0f19cc679eadd90e3e7e715033d4338744a076452f4d7a66056f4bce7e7c WatchSource:0}: Error finding container 93ec0f19cc679eadd90e3e7e715033d4338744a076452f4d7a66056f4bce7e7c: Status 404 returned error can't find the container with id 93ec0f19cc679eadd90e3e7e715033d4338744a076452f4d7a66056f4bce7e7c
Sep 09 06:03:59.205324 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:03:59.205286    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc" event={"ID":"34272d09-7edc-4e8e-95f4-a3a1466b6c2a","Type":"ContainerStarted","Data":"93ec0f19cc679eadd90e3e7e715033d4338744a076452f4d7a66056f4bce7e7c"}
Sep 09 06:04:00.213132 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:00.213039    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc" event={"ID":"34272d09-7edc-4e8e-95f4-a3a1466b6c2a","Type":"ContainerStarted","Data":"d15be9d54dd11f141da381e3c657530f2a298dc89f6dcf7029904fad0361be11"}
Sep 09 06:04:00.213132 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:00.213114    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:04:00.230973 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:00.230920    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc" podStartSLOduration=2.274130491 podStartE2EDuration="3.230903698s" totalImagesPullingTime="956.773207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:04:00.228516205 +0000 UTC m=+867.444846082" watchObservedRunningTime="2026-09-09 06:04:00.230903698 +0000 UTC m=+867.447233582"
Sep 09 06:04:06.224776 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:06.224746    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:04:06.994465 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:06.994426    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"]
Sep 09 06:04:06.998845 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:06.998817    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:07.001375 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.001351    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-f631ee-serving-cert\""
Sep 09 06:04:07.012454 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.012425    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"]
Sep 09 06:04:07.119479 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.119448    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a725293e-ba14-49f5-bbc3-c60f687559d7-metrics-certs\") pod \"maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:07.119655 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.119528    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a725293e-ba14-49f5-bbc3-c60f687559d7-maas-api-tls\") pod \"maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:07.119655 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.119570    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t9p8\" (UniqueName: \"kubernetes.io/projected/a725293e-ba14-49f5-bbc3-c60f687559d7-kube-api-access-6t9p8\") pod \"maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:07.119655 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.119598    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a725293e-ba14-49f5-bbc3-c60f687559d7-openshift-service-ca\") pod \"maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:07.221028 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.220997    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a725293e-ba14-49f5-bbc3-c60f687559d7-metrics-certs\") pod \"maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:07.221232 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.221048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a725293e-ba14-49f5-bbc3-c60f687559d7-maas-api-tls\") pod \"maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:07.221232 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:04:07.221149    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-serving-cert: secret "maas-api-e2e-explicit-f631ee-serving-cert" not found
Sep 09 06:04:07.221232 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:04:07.221200    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a725293e-ba14-49f5-bbc3-c60f687559d7-maas-api-tls podName:a725293e-ba14-49f5-bbc3-c60f687559d7 nodeName:}" failed. No retries permitted until 2026-09-09 06:04:07.721182916 +0000 UTC m=+874.937512782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a725293e-ba14-49f5-bbc3-c60f687559d7-maas-api-tls") pod "maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf" (UID: "a725293e-ba14-49f5-bbc3-c60f687559d7") : secret "maas-api-e2e-explicit-f631ee-serving-cert" not found
Sep 09 06:04:07.221415 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.221236    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t9p8\" (UniqueName: \"kubernetes.io/projected/a725293e-ba14-49f5-bbc3-c60f687559d7-kube-api-access-6t9p8\") pod \"maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:07.221415 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.221284    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a725293e-ba14-49f5-bbc3-c60f687559d7-openshift-service-ca\") pod \"maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:07.221957 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.221940    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a725293e-ba14-49f5-bbc3-c60f687559d7-openshift-service-ca\") pod \"maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:07.223552 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.223528    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a725293e-ba14-49f5-bbc3-c60f687559d7-metrics-certs\") pod \"maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:07.228954 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.228932    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t9p8\" (UniqueName: \"kubernetes.io/projected/a725293e-ba14-49f5-bbc3-c60f687559d7-kube-api-access-6t9p8\") pod \"maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:07.726812 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.726776    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a725293e-ba14-49f5-bbc3-c60f687559d7-maas-api-tls\") pod \"maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:07.729132 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.729113    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a725293e-ba14-49f5-bbc3-c60f687559d7-maas-api-tls\") pod \"maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:07.910409 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:07.910374    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:08.187044 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:08.187015    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"]
Sep 09 06:04:08.188012 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:04:08.187986    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda725293e_ba14_49f5_bbc3_c60f687559d7.slice/crio-93c879cae49c5e92c51c2f5587abeedf2fdc476cac9505f9dc20dc2e87ee4dcf WatchSource:0}: Error finding container 93c879cae49c5e92c51c2f5587abeedf2fdc476cac9505f9dc20dc2e87ee4dcf: Status 404 returned error can't find the container with id 93c879cae49c5e92c51c2f5587abeedf2fdc476cac9505f9dc20dc2e87ee4dcf
Sep 09 06:04:08.266646 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:08.266612    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf" event={"ID":"a725293e-ba14-49f5-bbc3-c60f687559d7","Type":"ContainerStarted","Data":"93c879cae49c5e92c51c2f5587abeedf2fdc476cac9505f9dc20dc2e87ee4dcf"}
Sep 09 06:04:09.275172 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:09.275136    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf" event={"ID":"a725293e-ba14-49f5-bbc3-c60f687559d7","Type":"ContainerStarted","Data":"8e7e883a9381eaf34fd8f61abfa8359c477fb0ad8422a44d4fd6df36188cb22d"}
Sep 09 06:04:09.275621 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:09.275197    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:09.293778 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:09.293727    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf" podStartSLOduration=2.877527907 podStartE2EDuration="3.293712676s" totalImagesPullingTime="416.184769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:04:09.291853727 +0000 UTC m=+876.508183624" watchObservedRunningTime="2026-09-09 06:04:09.293712676 +0000 UTC m=+876.510042560"
Sep 09 06:04:15.287277 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:15.287247    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:04:21.362934 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:21.362901    2564 generic.go:356] "Generic (PLEG): container finished" podID="9a0de45e-3b05-46a0-a50a-c5c62741425a" containerID="a3c0da4467a192963c2bf1e12f551d883bc960e62474911f4219c81ffb988be3" exitCode=0
Sep 09 06:04:21.363392 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:21.362959    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25" event={"ID":"9a0de45e-3b05-46a0-a50a-c5c62741425a","Type":"ContainerDied","Data":"a3c0da4467a192963c2bf1e12f551d883bc960e62474911f4219c81ffb988be3"}
Sep 09 06:04:21.363392 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:21.363356    2564 scope.go:117] "RemoveContainer" containerID="a3c0da4467a192963c2bf1e12f551d883bc960e62474911f4219c81ffb988be3"
Sep 09 06:04:21.364564 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:21.364533    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:04:21.364715 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:21.364612    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:04:21.364715 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:21.364658    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:04:22.371141 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:22.371107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25" event={"ID":"9a0de45e-3b05-46a0-a50a-c5c62741425a","Type":"ContainerStarted","Data":"a17fdd97585882b0cc9652bd6cac022fd68ba428c8b9fa6acf5396419cd1978e"}
Sep 09 06:04:23.085611 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:23.085573    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:04:23.086958 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:23.086912    2564 patch_prober.go:28] interesting pod/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 09 06:04:23.087067 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:23.086981    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25" podUID="9a0de45e-3b05-46a0-a50a-c5c62741425a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 09 06:04:24.085727 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:24.085688    2564 patch_prober.go:28] interesting pod/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 09 06:04:24.086101 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:24.085743    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25" podUID="9a0de45e-3b05-46a0-a50a-c5c62741425a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 09 06:04:25.085811 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:25.085769    2564 patch_prober.go:28] interesting pod/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 09 06:04:25.086282 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:25.085869    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25" podUID="9a0de45e-3b05-46a0-a50a-c5c62741425a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 09 06:04:26.085605 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:26.085478    2564 patch_prober.go:28] interesting pod/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 09 06:04:26.085789 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:26.085611    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25" podUID="9a0de45e-3b05-46a0-a50a-c5c62741425a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 09 06:04:26.185235 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:26.185199    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:04:27.085118 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:27.085081    2564 patch_prober.go:28] interesting pod/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 09 06:04:27.085298 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:27.085137    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25" podUID="9a0de45e-3b05-46a0-a50a-c5c62741425a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 09 06:04:28.085473 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.085433    2564 patch_prober.go:28] interesting pod/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 09 06:04:28.085865 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.085520    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25" podUID="9a0de45e-3b05-46a0-a50a-c5c62741425a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 09 06:04:28.417254 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.417168    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"]
Sep 09 06:04:28.417524 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.417478    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc" podUID="34272d09-7edc-4e8e-95f4-a3a1466b6c2a" containerName="maas-api" containerID="cri-o://d15be9d54dd11f141da381e3c657530f2a298dc89f6dcf7029904fad0361be11" gracePeriod=30
Sep 09 06:04:28.675577 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.675508    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:04:28.747313 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.747270    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-openshift-service-ca\") pod \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") "
Sep 09 06:04:28.747539 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.747330    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-metrics-certs\") pod \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") "
Sep 09 06:04:28.747539 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.747358    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-maas-api-tls\") pod \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") "
Sep 09 06:04:28.747539 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.747414    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgcqw\" (UniqueName: \"kubernetes.io/projected/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-kube-api-access-cgcqw\") pod \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\" (UID: \"34272d09-7edc-4e8e-95f4-a3a1466b6c2a\") "
Sep 09 06:04:28.747717 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.747676    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "34272d09-7edc-4e8e-95f4-a3a1466b6c2a" (UID: "34272d09-7edc-4e8e-95f4-a3a1466b6c2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:04:28.748051 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.747806    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:04:28.750129 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.750095    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "34272d09-7edc-4e8e-95f4-a3a1466b6c2a" (UID: "34272d09-7edc-4e8e-95f4-a3a1466b6c2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:04:28.750300 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.750281    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-kube-api-access-cgcqw" (OuterVolumeSpecName: "kube-api-access-cgcqw") pod "34272d09-7edc-4e8e-95f4-a3a1466b6c2a" (UID: "34272d09-7edc-4e8e-95f4-a3a1466b6c2a"). InnerVolumeSpecName "kube-api-access-cgcqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:04:28.750343 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.750292    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "34272d09-7edc-4e8e-95f4-a3a1466b6c2a" (UID: "34272d09-7edc-4e8e-95f4-a3a1466b6c2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:04:28.773804 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.773771    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"]
Sep 09 06:04:28.774061 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.774026    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25" podUID="9a0de45e-3b05-46a0-a50a-c5c62741425a" containerName="istio-proxy" containerID="cri-o://a17fdd97585882b0cc9652bd6cac022fd68ba428c8b9fa6acf5396419cd1978e" gracePeriod=30
Sep 09 06:04:28.848935 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.848896    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgcqw\" (UniqueName: \"kubernetes.io/projected/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-kube-api-access-cgcqw\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:04:28.848935 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.848926    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:04:28.848935 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:28.848935    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34272d09-7edc-4e8e-95f4-a3a1466b6c2a-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:04:29.425612 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:29.425577    2564 generic.go:356] "Generic (PLEG): container finished" podID="34272d09-7edc-4e8e-95f4-a3a1466b6c2a" containerID="d15be9d54dd11f141da381e3c657530f2a298dc89f6dcf7029904fad0361be11" exitCode=0
Sep 09 06:04:29.426006 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:29.425648    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"
Sep 09 06:04:29.426006 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:29.425668    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc" event={"ID":"34272d09-7edc-4e8e-95f4-a3a1466b6c2a","Type":"ContainerDied","Data":"d15be9d54dd11f141da381e3c657530f2a298dc89f6dcf7029904fad0361be11"}
Sep 09 06:04:29.426006 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:29.425725    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc" event={"ID":"34272d09-7edc-4e8e-95f4-a3a1466b6c2a","Type":"ContainerDied","Data":"93ec0f19cc679eadd90e3e7e715033d4338744a076452f4d7a66056f4bce7e7c"}
Sep 09 06:04:29.426006 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:29.425753    2564 scope.go:117] "RemoveContainer" containerID="d15be9d54dd11f141da381e3c657530f2a298dc89f6dcf7029904fad0361be11"
Sep 09 06:04:29.436021 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:29.436001    2564 scope.go:117] "RemoveContainer" containerID="d15be9d54dd11f141da381e3c657530f2a298dc89f6dcf7029904fad0361be11"
Sep 09 06:04:29.436318 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:04:29.436299    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d15be9d54dd11f141da381e3c657530f2a298dc89f6dcf7029904fad0361be11\": container with ID starting with d15be9d54dd11f141da381e3c657530f2a298dc89f6dcf7029904fad0361be11 not found: ID does not exist" containerID="d15be9d54dd11f141da381e3c657530f2a298dc89f6dcf7029904fad0361be11"
Sep 09 06:04:29.436361 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:29.436328    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d15be9d54dd11f141da381e3c657530f2a298dc89f6dcf7029904fad0361be11"} err="failed to get container status \"d15be9d54dd11f141da381e3c657530f2a298dc89f6dcf7029904fad0361be11\": rpc error: code = NotFound desc = could not find container \"d15be9d54dd11f141da381e3c657530f2a298dc89f6dcf7029904fad0361be11\": container with ID starting with d15be9d54dd11f141da381e3c657530f2a298dc89f6dcf7029904fad0361be11 not found: ID does not exist"
Sep 09 06:04:29.443913 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:29.443886    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"]
Sep 09 06:04:29.449230 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:29.449208    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d1d9fd16-5994bd945c-dtgrc"]
Sep 09 06:04:31.308943 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:31.308911    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34272d09-7edc-4e8e-95f4-a3a1466b6c2a" path="/var/lib/kubelet/pods/34272d09-7edc-4e8e-95f4-a3a1466b6c2a/volumes"
Sep 09 06:04:33.312599 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:33.312571    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 06:04:33.315641 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:33.315617    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 06:04:34.022268 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.022243    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:04:34.101583 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.101552    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-workload-socket\") pod \"9a0de45e-3b05-46a0-a50a-c5c62741425a\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") "
Sep 09 06:04:34.101766 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.101615    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0de45e-3b05-46a0-a50a-c5c62741425a-istiod-ca-cert\") pod \"9a0de45e-3b05-46a0-a50a-c5c62741425a\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") "
Sep 09 06:04:34.101766 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.101654    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-podinfo\") pod \"9a0de45e-3b05-46a0-a50a-c5c62741425a\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") "
Sep 09 06:04:34.101766 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.101679    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-credential-socket\") pod \"9a0de45e-3b05-46a0-a50a-c5c62741425a\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") "
Sep 09 06:04:34.101766 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.101725    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltdrp\" (UniqueName: \"kubernetes.io/projected/9a0de45e-3b05-46a0-a50a-c5c62741425a-kube-api-access-ltdrp\") pod \"9a0de45e-3b05-46a0-a50a-c5c62741425a\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") "
Sep 09 06:04:34.101985 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.101777    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-envoy\") pod \"9a0de45e-3b05-46a0-a50a-c5c62741425a\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") "
Sep 09 06:04:34.101985 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.101815    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-token\") pod \"9a0de45e-3b05-46a0-a50a-c5c62741425a\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") "
Sep 09 06:04:34.101985 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.101873    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9a0de45e-3b05-46a0-a50a-c5c62741425a" (UID: "9a0de45e-3b05-46a0-a50a-c5c62741425a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:04:34.101985 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.101954    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9a0de45e-3b05-46a0-a50a-c5c62741425a" (UID: "9a0de45e-3b05-46a0-a50a-c5c62741425a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:04:34.102219 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.102073    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0de45e-3b05-46a0-a50a-c5c62741425a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9a0de45e-3b05-46a0-a50a-c5c62741425a" (UID: "9a0de45e-3b05-46a0-a50a-c5c62741425a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:04:34.102219 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.102078    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-workload-certs\") pod \"9a0de45e-3b05-46a0-a50a-c5c62741425a\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") "
Sep 09 06:04:34.102219 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.102172    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-data\") pod \"9a0de45e-3b05-46a0-a50a-c5c62741425a\" (UID: \"9a0de45e-3b05-46a0-a50a-c5c62741425a\") "
Sep 09 06:04:34.102374 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.102263    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9a0de45e-3b05-46a0-a50a-c5c62741425a" (UID: "9a0de45e-3b05-46a0-a50a-c5c62741425a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:04:34.102458 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.102430    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-data" (OuterVolumeSpecName: "istio-data") pod "9a0de45e-3b05-46a0-a50a-c5c62741425a" (UID: "9a0de45e-3b05-46a0-a50a-c5c62741425a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:04:34.103960 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.103058    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-workload-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:04:34.103960 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.103085    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-data\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:04:34.103960 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.103109    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-workload-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:04:34.103960 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.103122    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0de45e-3b05-46a0-a50a-c5c62741425a-istiod-ca-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:04:34.103960 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.103135    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-credential-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:04:34.106001 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.105971    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9a0de45e-3b05-46a0-a50a-c5c62741425a" (UID: "9a0de45e-3b05-46a0-a50a-c5c62741425a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:04:34.106098 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.105975    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9a0de45e-3b05-46a0-a50a-c5c62741425a" (UID: "9a0de45e-3b05-46a0-a50a-c5c62741425a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 06:04:34.112177 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.112153    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0de45e-3b05-46a0-a50a-c5c62741425a-kube-api-access-ltdrp" (OuterVolumeSpecName: "kube-api-access-ltdrp") pod "9a0de45e-3b05-46a0-a50a-c5c62741425a" (UID: "9a0de45e-3b05-46a0-a50a-c5c62741425a"). InnerVolumeSpecName "kube-api-access-ltdrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:04:34.112285 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.112232    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a0de45e-3b05-46a0-a50a-c5c62741425a" (UID: "9a0de45e-3b05-46a0-a50a-c5c62741425a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:04:34.203880 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.203847    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltdrp\" (UniqueName: \"kubernetes.io/projected/9a0de45e-3b05-46a0-a50a-c5c62741425a-kube-api-access-ltdrp\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:04:34.203880 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.203875    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-envoy\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:04:34.203880 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.203884    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-token\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:04:34.204100 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.203893    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0de45e-3b05-46a0-a50a-c5c62741425a-istio-podinfo\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:04:34.462557 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.462465    2564 generic.go:356] "Generic (PLEG): container finished" podID="9a0de45e-3b05-46a0-a50a-c5c62741425a" containerID="a17fdd97585882b0cc9652bd6cac022fd68ba428c8b9fa6acf5396419cd1978e" exitCode=0
Sep 09 06:04:34.462557 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.462552    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"
Sep 09 06:04:34.463063 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.462549    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25" event={"ID":"9a0de45e-3b05-46a0-a50a-c5c62741425a","Type":"ContainerDied","Data":"a17fdd97585882b0cc9652bd6cac022fd68ba428c8b9fa6acf5396419cd1978e"}
Sep 09 06:04:34.463063 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.462615    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25" event={"ID":"9a0de45e-3b05-46a0-a50a-c5c62741425a","Type":"ContainerDied","Data":"3e6507bf5bee96ed73d15949e0af123eb2e39a835d904ae4c74f369e6420a19b"}
Sep 09 06:04:34.463063 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.462650    2564 scope.go:117] "RemoveContainer" containerID="a17fdd97585882b0cc9652bd6cac022fd68ba428c8b9fa6acf5396419cd1978e"
Sep 09 06:04:34.472710 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.472694    2564 scope.go:117] "RemoveContainer" containerID="a3c0da4467a192963c2bf1e12f551d883bc960e62474911f4219c81ffb988be3"
Sep 09 06:04:34.481408 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.481392    2564 scope.go:117] "RemoveContainer" containerID="a17fdd97585882b0cc9652bd6cac022fd68ba428c8b9fa6acf5396419cd1978e"
Sep 09 06:04:34.481673 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:04:34.481656    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17fdd97585882b0cc9652bd6cac022fd68ba428c8b9fa6acf5396419cd1978e\": container with ID starting with a17fdd97585882b0cc9652bd6cac022fd68ba428c8b9fa6acf5396419cd1978e not found: ID does not exist" containerID="a17fdd97585882b0cc9652bd6cac022fd68ba428c8b9fa6acf5396419cd1978e"
Sep 09 06:04:34.481713 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.481680    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17fdd97585882b0cc9652bd6cac022fd68ba428c8b9fa6acf5396419cd1978e"} err="failed to get container status \"a17fdd97585882b0cc9652bd6cac022fd68ba428c8b9fa6acf5396419cd1978e\": rpc error: code = NotFound desc = could not find container \"a17fdd97585882b0cc9652bd6cac022fd68ba428c8b9fa6acf5396419cd1978e\": container with ID starting with a17fdd97585882b0cc9652bd6cac022fd68ba428c8b9fa6acf5396419cd1978e not found: ID does not exist"
Sep 09 06:04:34.481713 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.481695    2564 scope.go:117] "RemoveContainer" containerID="a3c0da4467a192963c2bf1e12f551d883bc960e62474911f4219c81ffb988be3"
Sep 09 06:04:34.481919 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:04:34.481900    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3c0da4467a192963c2bf1e12f551d883bc960e62474911f4219c81ffb988be3\": container with ID starting with a3c0da4467a192963c2bf1e12f551d883bc960e62474911f4219c81ffb988be3 not found: ID does not exist" containerID="a3c0da4467a192963c2bf1e12f551d883bc960e62474911f4219c81ffb988be3"
Sep 09 06:04:34.481959 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.481925    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3c0da4467a192963c2bf1e12f551d883bc960e62474911f4219c81ffb988be3"} err="failed to get container status \"a3c0da4467a192963c2bf1e12f551d883bc960e62474911f4219c81ffb988be3\": rpc error: code = NotFound desc = could not find container \"a3c0da4467a192963c2bf1e12f551d883bc960e62474911f4219c81ffb988be3\": container with ID starting with a3c0da4467a192963c2bf1e12f551d883bc960e62474911f4219c81ffb988be3 not found: ID does not exist"
Sep 09 06:04:34.491757 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.491735    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"]
Sep 09 06:04:34.494315 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:34.494295    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d1d9fd16-openshift-default-589678f58d-7dt25"]
Sep 09 06:04:35.303637 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:04:35.303607    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a0de45e-3b05-46a0-a50a-c5c62741425a" path="/var/lib/kubelet/pods/9a0de45e-3b05-46a0-a50a-c5c62741425a/volumes"
Sep 09 06:05:45.630554 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.630523    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"]
Sep 09 06:05:45.630988 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.630960    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34272d09-7edc-4e8e-95f4-a3a1466b6c2a" containerName="maas-api"
Sep 09 06:05:45.630988 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.630973    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="34272d09-7edc-4e8e-95f4-a3a1466b6c2a" containerName="maas-api"
Sep 09 06:05:45.631071 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.630997    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a0de45e-3b05-46a0-a50a-c5c62741425a" containerName="istio-proxy"
Sep 09 06:05:45.631071 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.631003    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a0de45e-3b05-46a0-a50a-c5c62741425a" containerName="istio-proxy"
Sep 09 06:05:45.631136 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.631093    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a0de45e-3b05-46a0-a50a-c5c62741425a" containerName="istio-proxy"
Sep 09 06:05:45.631136 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.631104    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="34272d09-7edc-4e8e-95f4-a3a1466b6c2a" containerName="maas-api"
Sep 09 06:05:45.635461 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.635440    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.637833 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.637815    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fe8eaa\"/\"kube-root-ca.crt\""
Sep 09 06:05:45.637925 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.637814    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fe8eaa\"/\"openshift-service-ca.crt\""
Sep 09 06:05:45.638791 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.638765    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fe8eaa\"/\"default-dockercfg-r2hxt\""
Sep 09 06:05:45.638834 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.638779    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fe8eaa\"/\"test-model-fe8eaa-kserve-self-signed-certs\""
Sep 09 06:05:45.645397 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.645375    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"]
Sep 09 06:05:45.686843 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.686811    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-home\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.686960 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.686865    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-tls-certs\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.686960 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.686939    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-dshm\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.687099 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.686997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42tcj\" (UniqueName: \"kubernetes.io/projected/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-kube-api-access-42tcj\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.687099 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.687046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-model-cache\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.687099 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.687076    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-tmp-dir\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.787956 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.787921    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-tmp-dir\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.788113 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.787978    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-home\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.788113 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.788016    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-tls-certs\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.788113 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.788051    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-dshm\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.788113 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.788082    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42tcj\" (UniqueName: \"kubernetes.io/projected/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-kube-api-access-42tcj\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.788308 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.788217    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-model-cache\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.788398 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.788376    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-tmp-dir\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.788398 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.788390    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-home\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.788578 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.788560    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-model-cache\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.790317 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.790294    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-dshm\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.790601 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.790584    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-tls-certs\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.797884 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.797808    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42tcj\" (UniqueName: \"kubernetes.io/projected/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-kube-api-access-42tcj\") pod \"test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") " pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:45.946101 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:45.946012    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:46.220073 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:46.220048    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"]
Sep 09 06:05:46.221996 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:05:46.221965    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1182f86e_73cd_48a6_8ac3_f4f9a80af43e.slice/crio-fb26e16da5ad8ef4faa52abc7a5fc0e46ffd1afaf8fdc6cfd10fc880fb84e62d WatchSource:0}: Error finding container fb26e16da5ad8ef4faa52abc7a5fc0e46ffd1afaf8fdc6cfd10fc880fb84e62d: Status 404 returned error can't find the container with id fb26e16da5ad8ef4faa52abc7a5fc0e46ffd1afaf8fdc6cfd10fc880fb84e62d
Sep 09 06:05:46.906256 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:46.906221    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh" event={"ID":"1182f86e-73cd-48a6-8ac3-f4f9a80af43e","Type":"ContainerStarted","Data":"a192b6094b38a3ad93aa8547b998e0cf7f943269d293184d524a4aece9ef0e17"}
Sep 09 06:05:46.906256 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:46.906262    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh" event={"ID":"1182f86e-73cd-48a6-8ac3-f4f9a80af43e","Type":"ContainerStarted","Data":"fb26e16da5ad8ef4faa52abc7a5fc0e46ffd1afaf8fdc6cfd10fc880fb84e62d"}
Sep 09 06:05:46.923537 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:46.923467    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh" podStartSLOduration=1.9234540820000001 podStartE2EDuration="1.923454082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:05:46.923176913 +0000 UTC m=+974.139506799" watchObservedRunningTime="2026-09-09 06:05:46.923454082 +0000 UTC m=+974.139783966"
Sep 09 06:05:53.884989 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:53.884943    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"]
Sep 09 06:05:53.885432 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:53.885187    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf" podUID="a725293e-ba14-49f5-bbc3-c60f687559d7" containerName="maas-api" containerID="cri-o://8e7e883a9381eaf34fd8f61abfa8359c477fb0ad8422a44d4fd6df36188cb22d" gracePeriod=30
Sep 09 06:05:54.150006 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.149940    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:05:54.277866 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.277834    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a725293e-ba14-49f5-bbc3-c60f687559d7-maas-api-tls\") pod \"a725293e-ba14-49f5-bbc3-c60f687559d7\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") "
Sep 09 06:05:54.278027 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.277913    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a725293e-ba14-49f5-bbc3-c60f687559d7-metrics-certs\") pod \"a725293e-ba14-49f5-bbc3-c60f687559d7\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") "
Sep 09 06:05:54.278102 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.278079    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t9p8\" (UniqueName: \"kubernetes.io/projected/a725293e-ba14-49f5-bbc3-c60f687559d7-kube-api-access-6t9p8\") pod \"a725293e-ba14-49f5-bbc3-c60f687559d7\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") "
Sep 09 06:05:54.278199 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.278180    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a725293e-ba14-49f5-bbc3-c60f687559d7-openshift-service-ca\") pod \"a725293e-ba14-49f5-bbc3-c60f687559d7\" (UID: \"a725293e-ba14-49f5-bbc3-c60f687559d7\") "
Sep 09 06:05:54.278607 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.278576    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a725293e-ba14-49f5-bbc3-c60f687559d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a725293e-ba14-49f5-bbc3-c60f687559d7" (UID: "a725293e-ba14-49f5-bbc3-c60f687559d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:05:54.280095 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.280058    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a725293e-ba14-49f5-bbc3-c60f687559d7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a725293e-ba14-49f5-bbc3-c60f687559d7" (UID: "a725293e-ba14-49f5-bbc3-c60f687559d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:05:54.280202 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.280122    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a725293e-ba14-49f5-bbc3-c60f687559d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a725293e-ba14-49f5-bbc3-c60f687559d7" (UID: "a725293e-ba14-49f5-bbc3-c60f687559d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:05:54.280303 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.280282    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a725293e-ba14-49f5-bbc3-c60f687559d7-kube-api-access-6t9p8" (OuterVolumeSpecName: "kube-api-access-6t9p8") pod "a725293e-ba14-49f5-bbc3-c60f687559d7" (UID: "a725293e-ba14-49f5-bbc3-c60f687559d7"). InnerVolumeSpecName "kube-api-access-6t9p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:05:54.380548 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.380077    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a725293e-ba14-49f5-bbc3-c60f687559d7-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:05:54.380548 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.380118    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a725293e-ba14-49f5-bbc3-c60f687559d7-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:05:54.380548 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.380132    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a725293e-ba14-49f5-bbc3-c60f687559d7-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:05:54.380548 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.380146    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t9p8\" (UniqueName: \"kubernetes.io/projected/a725293e-ba14-49f5-bbc3-c60f687559d7-kube-api-access-6t9p8\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:05:54.960599 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.960565    2564 generic.go:356] "Generic (PLEG): container finished" podID="a725293e-ba14-49f5-bbc3-c60f687559d7" containerID="8e7e883a9381eaf34fd8f61abfa8359c477fb0ad8422a44d4fd6df36188cb22d" exitCode=0
Sep 09 06:05:54.961087 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.960622    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"
Sep 09 06:05:54.961087 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.960652    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf" event={"ID":"a725293e-ba14-49f5-bbc3-c60f687559d7","Type":"ContainerDied","Data":"8e7e883a9381eaf34fd8f61abfa8359c477fb0ad8422a44d4fd6df36188cb22d"}
Sep 09 06:05:54.961087 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.960721    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf" event={"ID":"a725293e-ba14-49f5-bbc3-c60f687559d7","Type":"ContainerDied","Data":"93c879cae49c5e92c51c2f5587abeedf2fdc476cac9505f9dc20dc2e87ee4dcf"}
Sep 09 06:05:54.961087 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.960762    2564 scope.go:117] "RemoveContainer" containerID="8e7e883a9381eaf34fd8f61abfa8359c477fb0ad8422a44d4fd6df36188cb22d"
Sep 09 06:05:54.970730 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.970714    2564 scope.go:117] "RemoveContainer" containerID="8e7e883a9381eaf34fd8f61abfa8359c477fb0ad8422a44d4fd6df36188cb22d"
Sep 09 06:05:54.970998 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:05:54.970979    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7e883a9381eaf34fd8f61abfa8359c477fb0ad8422a44d4fd6df36188cb22d\": container with ID starting with 8e7e883a9381eaf34fd8f61abfa8359c477fb0ad8422a44d4fd6df36188cb22d not found: ID does not exist" containerID="8e7e883a9381eaf34fd8f61abfa8359c477fb0ad8422a44d4fd6df36188cb22d"
Sep 09 06:05:54.971045 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.971008    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7e883a9381eaf34fd8f61abfa8359c477fb0ad8422a44d4fd6df36188cb22d"} err="failed to get container status \"8e7e883a9381eaf34fd8f61abfa8359c477fb0ad8422a44d4fd6df36188cb22d\": rpc error: code = NotFound desc = could not find container \"8e7e883a9381eaf34fd8f61abfa8359c477fb0ad8422a44d4fd6df36188cb22d\": container with ID starting with 8e7e883a9381eaf34fd8f61abfa8359c477fb0ad8422a44d4fd6df36188cb22d not found: ID does not exist"
Sep 09 06:05:54.981768 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.981747    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"]
Sep 09 06:05:54.985706 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:54.985686    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f631ee-5d5cd69c4c-4qztf"]
Sep 09 06:05:55.302917 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:55.302835    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a725293e-ba14-49f5-bbc3-c60f687559d7" path="/var/lib/kubelet/pods/a725293e-ba14-49f5-bbc3-c60f687559d7/volumes"
Sep 09 06:05:55.946563 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:55.946458    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:55.946563 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:55.946515    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:55.959482 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:55.959458    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:05:55.979600 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:05:55.979568    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:06:07.818547 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:07.818512    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"]
Sep 09 06:06:07.819108 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:07.819064    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a725293e-ba14-49f5-bbc3-c60f687559d7" containerName="maas-api"
Sep 09 06:06:07.819108 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:07.819080    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a725293e-ba14-49f5-bbc3-c60f687559d7" containerName="maas-api"
Sep 09 06:06:07.819108 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:07.819089    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a0de45e-3b05-46a0-a50a-c5c62741425a" containerName="istio-proxy"
Sep 09 06:06:07.819108 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:07.819095    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a0de45e-3b05-46a0-a50a-c5c62741425a" containerName="istio-proxy"
Sep 09 06:06:07.819337 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:07.819193    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a0de45e-3b05-46a0-a50a-c5c62741425a" containerName="istio-proxy"
Sep 09 06:06:07.819337 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:07.819203    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a725293e-ba14-49f5-bbc3-c60f687559d7" containerName="maas-api"
Sep 09 06:06:07.822477 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:07.822456    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:07.824934 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:07.824909    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-464e41e7-serving-cert\""
Sep 09 06:06:07.838284 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:07.831978    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"]
Sep 09 06:06:07.911096 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:07.911057    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-metrics-certs\") pod \"maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:07.911270 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:07.911128    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-maas-api-tls\") pod \"maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:07.911270 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:07.911189    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-openshift-service-ca\") pod \"maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:07.911270 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:07.911233    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxkp8\" (UniqueName: \"kubernetes.io/projected/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-kube-api-access-nxkp8\") pod \"maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:08.012740 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:08.012705    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-metrics-certs\") pod \"maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:08.012912 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:08.012756    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-maas-api-tls\") pod \"maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:08.012912 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:08.012801    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-openshift-service-ca\") pod \"maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:08.012912 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:08.012818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxkp8\" (UniqueName: \"kubernetes.io/projected/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-kube-api-access-nxkp8\") pod \"maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:08.013508 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:08.013467    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-openshift-service-ca\") pod \"maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:08.015260 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:08.015237    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-metrics-certs\") pod \"maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:08.015355 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:08.015278    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-maas-api-tls\") pod \"maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:08.020845 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:08.020819    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxkp8\" (UniqueName: \"kubernetes.io/projected/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-kube-api-access-nxkp8\") pod \"maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:08.151877 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:08.151836    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:08.441278 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:08.441252    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"]
Sep 09 06:06:08.442208 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:06:08.442180    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12acf1fd_495d_4e2a_b95f_f75baaf95a9a.slice/crio-d083a972a7f8559e46f712b8b45012bf12e9e13336541fc8af224bcf6416292b WatchSource:0}: Error finding container d083a972a7f8559e46f712b8b45012bf12e9e13336541fc8af224bcf6416292b: Status 404 returned error can't find the container with id d083a972a7f8559e46f712b8b45012bf12e9e13336541fc8af224bcf6416292b
Sep 09 06:06:08.443781 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:08.443763    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 06:06:09.050426 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:09.050392    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9" event={"ID":"12acf1fd-495d-4e2a-b95f-f75baaf95a9a","Type":"ContainerStarted","Data":"d083a972a7f8559e46f712b8b45012bf12e9e13336541fc8af224bcf6416292b"}
Sep 09 06:06:11.065065 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:11.064965    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9" event={"ID":"12acf1fd-495d-4e2a-b95f-f75baaf95a9a","Type":"ContainerStarted","Data":"bbbd98a36742066a06b08061719d7cca81f1106bcfdf2bc050ba1c2bf0521e26"}
Sep 09 06:06:11.065436 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:11.065132    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:11.087980 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:11.087923    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9" podStartSLOduration=2.36987568 podStartE2EDuration="4.087909598s" totalImagesPullingTime="1.718033918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:06:11.085058722 +0000 UTC m=+998.301388606" watchObservedRunningTime="2026-09-09 06:06:11.087909598 +0000 UTC m=+998.304239499"
Sep 09 06:06:17.079673 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:17.079641    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:49.206892 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:49.206857    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"]
Sep 09 06:06:49.207442 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:49.207133    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9" podUID="12acf1fd-495d-4e2a-b95f-f75baaf95a9a" containerName="maas-api" containerID="cri-o://bbbd98a36742066a06b08061719d7cca81f1106bcfdf2bc050ba1c2bf0521e26" gracePeriod=30
Sep 09 06:06:49.468557 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:49.468444    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:49.535208 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:49.535172    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxkp8\" (UniqueName: \"kubernetes.io/projected/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-kube-api-access-nxkp8\") pod \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") "
Sep 09 06:06:49.535208 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:49.535212    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-maas-api-tls\") pod \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") "
Sep 09 06:06:49.535410 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:49.535237    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-metrics-certs\") pod \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") "
Sep 09 06:06:49.535410 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:49.535367    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-openshift-service-ca\") pod \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\" (UID: \"12acf1fd-495d-4e2a-b95f-f75baaf95a9a\") "
Sep 09 06:06:49.535795 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:49.535766    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12acf1fd-495d-4e2a-b95f-f75baaf95a9a" (UID: "12acf1fd-495d-4e2a-b95f-f75baaf95a9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:06:49.537348 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:49.537322    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12acf1fd-495d-4e2a-b95f-f75baaf95a9a" (UID: "12acf1fd-495d-4e2a-b95f-f75baaf95a9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:06:49.537466 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:49.537371    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "12acf1fd-495d-4e2a-b95f-f75baaf95a9a" (UID: "12acf1fd-495d-4e2a-b95f-f75baaf95a9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:06:49.537466 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:49.537401    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-kube-api-access-nxkp8" (OuterVolumeSpecName: "kube-api-access-nxkp8") pod "12acf1fd-495d-4e2a-b95f-f75baaf95a9a" (UID: "12acf1fd-495d-4e2a-b95f-f75baaf95a9a"). InnerVolumeSpecName "kube-api-access-nxkp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:06:49.636832 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:49.636795    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxkp8\" (UniqueName: \"kubernetes.io/projected/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-kube-api-access-nxkp8\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:06:49.636832 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:49.636827    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:06:49.636832 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:49.636840    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:06:49.637066 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:49.636851    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12acf1fd-495d-4e2a-b95f-f75baaf95a9a-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:06:50.314847 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:50.314811    2564 generic.go:356] "Generic (PLEG): container finished" podID="12acf1fd-495d-4e2a-b95f-f75baaf95a9a" containerID="bbbd98a36742066a06b08061719d7cca81f1106bcfdf2bc050ba1c2bf0521e26" exitCode=0
Sep 09 06:06:50.315251 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:50.314852    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"
Sep 09 06:06:50.315251 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:50.314892    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9" event={"ID":"12acf1fd-495d-4e2a-b95f-f75baaf95a9a","Type":"ContainerDied","Data":"bbbd98a36742066a06b08061719d7cca81f1106bcfdf2bc050ba1c2bf0521e26"}
Sep 09 06:06:50.315251 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:50.314944    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9" event={"ID":"12acf1fd-495d-4e2a-b95f-f75baaf95a9a","Type":"ContainerDied","Data":"d083a972a7f8559e46f712b8b45012bf12e9e13336541fc8af224bcf6416292b"}
Sep 09 06:06:50.315251 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:50.314968    2564 scope.go:117] "RemoveContainer" containerID="bbbd98a36742066a06b08061719d7cca81f1106bcfdf2bc050ba1c2bf0521e26"
Sep 09 06:06:50.324990 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:50.324973    2564 scope.go:117] "RemoveContainer" containerID="bbbd98a36742066a06b08061719d7cca81f1106bcfdf2bc050ba1c2bf0521e26"
Sep 09 06:06:50.325241 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:06:50.325225    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbbd98a36742066a06b08061719d7cca81f1106bcfdf2bc050ba1c2bf0521e26\": container with ID starting with bbbd98a36742066a06b08061719d7cca81f1106bcfdf2bc050ba1c2bf0521e26 not found: ID does not exist" containerID="bbbd98a36742066a06b08061719d7cca81f1106bcfdf2bc050ba1c2bf0521e26"
Sep 09 06:06:50.325281 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:50.325249    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbbd98a36742066a06b08061719d7cca81f1106bcfdf2bc050ba1c2bf0521e26"} err="failed to get container status \"bbbd98a36742066a06b08061719d7cca81f1106bcfdf2bc050ba1c2bf0521e26\": rpc error: code = NotFound desc = could not find container \"bbbd98a36742066a06b08061719d7cca81f1106bcfdf2bc050ba1c2bf0521e26\": container with ID starting with bbbd98a36742066a06b08061719d7cca81f1106bcfdf2bc050ba1c2bf0521e26 not found: ID does not exist"
Sep 09 06:06:50.337033 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:50.337009    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"]
Sep 09 06:06:50.344430 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:50.344406    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464e41e7-58f9b47894-hxlb9"]
Sep 09 06:06:51.302934 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:06:51.302904    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12acf1fd-495d-4e2a-b95f-f75baaf95a9a" path="/var/lib/kubelet/pods/12acf1fd-495d-4e2a-b95f-f75baaf95a9a/volumes"
Sep 09 06:07:13.860067 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:13.860027    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"]
Sep 09 06:07:13.860578 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:13.860474    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12acf1fd-495d-4e2a-b95f-f75baaf95a9a" containerName="maas-api"
Sep 09 06:07:13.860578 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:13.860486    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="12acf1fd-495d-4e2a-b95f-f75baaf95a9a" containerName="maas-api"
Sep 09 06:07:13.860656 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:13.860602    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="12acf1fd-495d-4e2a-b95f-f75baaf95a9a" containerName="maas-api"
Sep 09 06:07:13.863906 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:13.863888    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:13.866166 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:13.866141    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-f3d736-serving-cert\""
Sep 09 06:07:13.877196 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:13.877168    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"]
Sep 09 06:07:13.979047 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:13.979002    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg745\" (UniqueName: \"kubernetes.io/projected/f1c34dc6-f827-420b-aa07-0ceb7535dad5-kube-api-access-vg745\") pod \"maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:13.979336 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:13.979314    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c34dc6-f827-420b-aa07-0ceb7535dad5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:13.979579 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:13.979551    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c34dc6-f827-420b-aa07-0ceb7535dad5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:13.979732 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:13.979704    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c34dc6-f827-420b-aa07-0ceb7535dad5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:14.080599 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:14.080565    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c34dc6-f827-420b-aa07-0ceb7535dad5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:14.080785 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:14.080630    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg745\" (UniqueName: \"kubernetes.io/projected/f1c34dc6-f827-420b-aa07-0ceb7535dad5-kube-api-access-vg745\") pod \"maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:14.080785 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:14.080664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c34dc6-f827-420b-aa07-0ceb7535dad5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:14.080785 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:14.080695    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c34dc6-f827-420b-aa07-0ceb7535dad5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:14.081402 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:14.081376    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c34dc6-f827-420b-aa07-0ceb7535dad5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:14.083057 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:14.083031    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c34dc6-f827-420b-aa07-0ceb7535dad5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:14.083144 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:14.083079    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c34dc6-f827-420b-aa07-0ceb7535dad5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:14.087758 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:14.087737    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg745\" (UniqueName: \"kubernetes.io/projected/f1c34dc6-f827-420b-aa07-0ceb7535dad5-kube-api-access-vg745\") pod \"maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:14.179201 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:14.179114    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:14.454252 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:14.454225    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"]
Sep 09 06:07:14.455854 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:07:14.455828    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c34dc6_f827_420b_aa07_0ceb7535dad5.slice/crio-346e0ed9ac12de89efa100233df2e4b8dcb1b74cf91d7e2ac5416b405a3a894c WatchSource:0}: Error finding container 346e0ed9ac12de89efa100233df2e4b8dcb1b74cf91d7e2ac5416b405a3a894c: Status 404 returned error can't find the container with id 346e0ed9ac12de89efa100233df2e4b8dcb1b74cf91d7e2ac5416b405a3a894c
Sep 09 06:07:14.466784 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:14.466755    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc" event={"ID":"f1c34dc6-f827-420b-aa07-0ceb7535dad5","Type":"ContainerStarted","Data":"346e0ed9ac12de89efa100233df2e4b8dcb1b74cf91d7e2ac5416b405a3a894c"}
Sep 09 06:07:15.474825 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:15.474787    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc" event={"ID":"f1c34dc6-f827-420b-aa07-0ceb7535dad5","Type":"ContainerStarted","Data":"96c9ecae783a4938d2f7823ed0a797fd446d3e28edb7040a3fddc3ca063ba610"}
Sep 09 06:07:15.475202 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:15.474881    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:15.496405 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:15.496358    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc" podStartSLOduration=1.998707534 podStartE2EDuration="2.49634424s" totalImagesPullingTime="497.636706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:07:15.49433094 +0000 UTC m=+1062.710660825" watchObservedRunningTime="2026-09-09 06:07:15.49634424 +0000 UTC m=+1062.712674123"
Sep 09 06:07:21.486193 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:21.486159    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:07:38.337556 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.337521    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"]
Sep 09 06:07:38.342536 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.342512    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.345135 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.345113    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-750c68-openshift-default-dockercfg-br9lj\""
Sep 09 06:07:38.353062 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.352574    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"]
Sep 09 06:07:38.411612 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.411580    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-envoy\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.411795 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.411658    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-podinfo\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.411795 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.411697    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-data\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.411795 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.411767    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-token\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.411927 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.411792    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wmzw\" (UniqueName: \"kubernetes.io/projected/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-kube-api-access-9wmzw\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.411927 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.411850    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-workload-socket\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.411927 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.411883    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istiod-ca-cert\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.411927 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.411903    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-workload-certs\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.411927 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.411921    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-credential-socket\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.513288 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.513252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-credential-socket\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.513465 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.513410    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-envoy\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.513465 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.513441    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-podinfo\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.513613 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.513466    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-data\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.513673 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.513612    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-token\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.513673 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.513649    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wmzw\" (UniqueName: \"kubernetes.io/projected/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-kube-api-access-9wmzw\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.513788 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.513690    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-workload-socket\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.513788 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.513698    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-credential-socket\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.513788 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.513724    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istiod-ca-cert\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.513788 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.513756    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-workload-certs\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.513788 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.513779    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-data\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.514052 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.513957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-workload-socket\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.514143 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.514122    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-workload-certs\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.514388 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.514368    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istiod-ca-cert\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.515818 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.515796    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-envoy\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.515915 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.515856    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-podinfo\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.521086 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.521060    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-token\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.521170 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.521115    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wmzw\" (UniqueName: \"kubernetes.io/projected/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-kube-api-access-9wmzw\") pod \"e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") " pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.658283 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.658201    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:38.940873 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.940839    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"]
Sep 09 06:07:38.941017 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:07:38.940999    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3eeac3_b584_4fb6_8c0d_0cb42ad3152f.slice/crio-20384d97911aab4d2fe915e3cba8b5d46af706278467a7aff524d5a558fc388d WatchSource:0}: Error finding container 20384d97911aab4d2fe915e3cba8b5d46af706278467a7aff524d5a558fc388d: Status 404 returned error can't find the container with id 20384d97911aab4d2fe915e3cba8b5d46af706278467a7aff524d5a558fc388d
Sep 09 06:07:38.943339 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.943307    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:07:38.943435 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.943366    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:07:38.943435 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:38.943394    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:07:39.622613 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:39.622575    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj" event={"ID":"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f","Type":"ContainerStarted","Data":"11cef41c04716d70dd62e77976c5adf97318fd77e1db4ea87bc72b1b3c308e06"}
Sep 09 06:07:39.623084 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:39.622632    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj" event={"ID":"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f","Type":"ContainerStarted","Data":"20384d97911aab4d2fe915e3cba8b5d46af706278467a7aff524d5a558fc388d"}
Sep 09 06:07:39.644099 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:39.644048    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj" podStartSLOduration=1.6440321039999999 podStartE2EDuration="1.644032104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:07:39.641050263 +0000 UTC m=+1086.857380147" watchObservedRunningTime="2026-09-09 06:07:39.644032104 +0000 UTC m=+1086.860361987"
Sep 09 06:07:39.659040 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:39.659010    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:40.664962 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:40.664933    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:41.634627 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:41.634591    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:41.635630 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:41.635608    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:07:42.649197 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:42.649167    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"]
Sep 09 06:07:42.653008 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:42.652988    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:42.656045 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:42.656013    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-750c68-serving-cert\""
Sep 09 06:07:42.659397 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:42.659316    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"]
Sep 09 06:07:42.753158 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:42.753124    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321b3854-6df4-469f-8b5a-c75de70c685e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:42.753346 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:42.753300    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321b3854-6df4-469f-8b5a-c75de70c685e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:42.753407 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:42.753365    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321b3854-6df4-469f-8b5a-c75de70c685e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:42.753455 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:42.753433    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f4qd\" (UniqueName: \"kubernetes.io/projected/321b3854-6df4-469f-8b5a-c75de70c685e-kube-api-access-5f4qd\") pod \"maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:42.854571 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:42.854535    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321b3854-6df4-469f-8b5a-c75de70c685e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:42.854760 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:42.854640    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321b3854-6df4-469f-8b5a-c75de70c685e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:42.854760 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:42.854663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321b3854-6df4-469f-8b5a-c75de70c685e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:42.854760 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:07:42.854689    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-serving-cert: secret "maas-api-e2e-shared-b-w2-750c68-serving-cert" not found
Sep 09 06:07:42.854760 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:42.854694    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f4qd\" (UniqueName: \"kubernetes.io/projected/321b3854-6df4-469f-8b5a-c75de70c685e-kube-api-access-5f4qd\") pod \"maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:42.854979 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:07:42.854763    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321b3854-6df4-469f-8b5a-c75de70c685e-maas-api-tls podName:321b3854-6df4-469f-8b5a-c75de70c685e nodeName:}" failed. No retries permitted until 2026-09-09 06:07:43.35474106 +0000 UTC m=+1090.571070922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/321b3854-6df4-469f-8b5a-c75de70c685e-maas-api-tls") pod "maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9" (UID: "321b3854-6df4-469f-8b5a-c75de70c685e") : secret "maas-api-e2e-shared-b-w2-750c68-serving-cert" not found
Sep 09 06:07:42.855449 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:42.855422    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321b3854-6df4-469f-8b5a-c75de70c685e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:42.857152 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:42.857133    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321b3854-6df4-469f-8b5a-c75de70c685e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:42.863041 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:42.863024    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f4qd\" (UniqueName: \"kubernetes.io/projected/321b3854-6df4-469f-8b5a-c75de70c685e-kube-api-access-5f4qd\") pod \"maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:43.359627 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:43.359592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321b3854-6df4-469f-8b5a-c75de70c685e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:43.362179 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:43.362151    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321b3854-6df4-469f-8b5a-c75de70c685e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:43.567468 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:43.567437    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:43.839989 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:43.839959    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"]
Sep 09 06:07:43.840530 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:07:43.840469    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321b3854_6df4_469f_8b5a_c75de70c685e.slice/crio-b1901a7e4abaab91d9f2a967c7351dcbfd4f1ae8cc373febf3ecf63473cb973d WatchSource:0}: Error finding container b1901a7e4abaab91d9f2a967c7351dcbfd4f1ae8cc373febf3ecf63473cb973d: Status 404 returned error can't find the container with id b1901a7e4abaab91d9f2a967c7351dcbfd4f1ae8cc373febf3ecf63473cb973d
Sep 09 06:07:44.663553 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:44.663512    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9" event={"ID":"321b3854-6df4-469f-8b5a-c75de70c685e","Type":"ContainerStarted","Data":"5a2dbe4ba3f16b98737a9598dcbbd526daf7ecbbd0aa8c02b5aa05983093ee47"}
Sep 09 06:07:44.663763 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:44.663579    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9" event={"ID":"321b3854-6df4-469f-8b5a-c75de70c685e","Type":"ContainerStarted","Data":"b1901a7e4abaab91d9f2a967c7351dcbfd4f1ae8cc373febf3ecf63473cb973d"}
Sep 09 06:07:44.663763 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:44.663606    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:44.682660 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:44.682612    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9" podStartSLOduration=2.355870596 podStartE2EDuration="2.682594942s" totalImagesPullingTime="326.724346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:07:44.680147796 +0000 UTC m=+1091.896477683" watchObservedRunningTime="2026-09-09 06:07:44.682594942 +0000 UTC m=+1091.898924826"
Sep 09 06:07:50.675383 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:50.675355    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:07:53.111126 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:07:53.111090    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"]
Sep 09 06:08:01.053147 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:01.053101    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"]
Sep 09 06:08:08.131254 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.131215    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z" podUID="0da978ab-db7e-44ea-893e-cd60c07bae07" containerName="main" containerID="cri-o://413db7d5fa1756bfe36d375d27b2c80d9d0568eaaaf688e1ebb7b5d54bfc938f" gracePeriod=45
Sep 09 06:08:08.448007 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.447666    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:08:08.513110 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.513078    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-tmp-dir\") pod \"0da978ab-db7e-44ea-893e-cd60c07bae07\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") "
Sep 09 06:08:08.513303 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.513135    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da978ab-db7e-44ea-893e-cd60c07bae07-tls-certs\") pod \"0da978ab-db7e-44ea-893e-cd60c07bae07\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") "
Sep 09 06:08:08.513303 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.513158    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-dshm\") pod \"0da978ab-db7e-44ea-893e-cd60c07bae07\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") "
Sep 09 06:08:08.513303 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.513176    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgbrg\" (UniqueName: \"kubernetes.io/projected/0da978ab-db7e-44ea-893e-cd60c07bae07-kube-api-access-kgbrg\") pod \"0da978ab-db7e-44ea-893e-cd60c07bae07\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") "
Sep 09 06:08:08.513303 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.513256    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-home\") pod \"0da978ab-db7e-44ea-893e-cd60c07bae07\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") "
Sep 09 06:08:08.513525 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.513344    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-model-cache\") pod \"0da978ab-db7e-44ea-893e-cd60c07bae07\" (UID: \"0da978ab-db7e-44ea-893e-cd60c07bae07\") "
Sep 09 06:08:08.513525 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.513374    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0da978ab-db7e-44ea-893e-cd60c07bae07" (UID: "0da978ab-db7e-44ea-893e-cd60c07bae07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:08:08.513707 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.513664    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-home" (OuterVolumeSpecName: "home") pod "0da978ab-db7e-44ea-893e-cd60c07bae07" (UID: "0da978ab-db7e-44ea-893e-cd60c07bae07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:08:08.513707 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.513708    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-tmp-dir\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:08.514004 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.513980    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-model-cache" (OuterVolumeSpecName: "model-cache") pod "0da978ab-db7e-44ea-893e-cd60c07bae07" (UID: "0da978ab-db7e-44ea-893e-cd60c07bae07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:08:08.515410 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.515382    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-dshm" (OuterVolumeSpecName: "dshm") pod "0da978ab-db7e-44ea-893e-cd60c07bae07" (UID: "0da978ab-db7e-44ea-893e-cd60c07bae07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:08:08.515410 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.515406    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da978ab-db7e-44ea-893e-cd60c07bae07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0da978ab-db7e-44ea-893e-cd60c07bae07" (UID: "0da978ab-db7e-44ea-893e-cd60c07bae07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:08:08.515599 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.515527    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da978ab-db7e-44ea-893e-cd60c07bae07-kube-api-access-kgbrg" (OuterVolumeSpecName: "kube-api-access-kgbrg") pod "0da978ab-db7e-44ea-893e-cd60c07bae07" (UID: "0da978ab-db7e-44ea-893e-cd60c07bae07"). InnerVolumeSpecName "kube-api-access-kgbrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:08:08.614768 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.614734    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-dshm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:08.614768 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.614762    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgbrg\" (UniqueName: \"kubernetes.io/projected/0da978ab-db7e-44ea-893e-cd60c07bae07-kube-api-access-kgbrg\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:08.614768 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.614772    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-home\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:08.615003 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.614781    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da978ab-db7e-44ea-893e-cd60c07bae07-model-cache\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:08.615003 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.614789    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da978ab-db7e-44ea-893e-cd60c07bae07-tls-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:08.818294 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.818214    2564 generic.go:356] "Generic (PLEG): container finished" podID="0da978ab-db7e-44ea-893e-cd60c07bae07" containerID="413db7d5fa1756bfe36d375d27b2c80d9d0568eaaaf688e1ebb7b5d54bfc938f" exitCode=0
Sep 09 06:08:08.818294 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.818253    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"
Sep 09 06:08:08.818473 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.818256    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z" event={"ID":"0da978ab-db7e-44ea-893e-cd60c07bae07","Type":"ContainerDied","Data":"413db7d5fa1756bfe36d375d27b2c80d9d0568eaaaf688e1ebb7b5d54bfc938f"}
Sep 09 06:08:08.818473 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.818367    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z" event={"ID":"0da978ab-db7e-44ea-893e-cd60c07bae07","Type":"ContainerDied","Data":"1dad9c80cb6689700eaa3f8118636645e6d47b926dbb2dd7fe99fd160efadf80"}
Sep 09 06:08:08.818473 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.818393    2564 scope.go:117] "RemoveContainer" containerID="413db7d5fa1756bfe36d375d27b2c80d9d0568eaaaf688e1ebb7b5d54bfc938f"
Sep 09 06:08:08.827977 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.827962    2564 scope.go:117] "RemoveContainer" containerID="413db7d5fa1756bfe36d375d27b2c80d9d0568eaaaf688e1ebb7b5d54bfc938f"
Sep 09 06:08:08.828270 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:08:08.828249    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"413db7d5fa1756bfe36d375d27b2c80d9d0568eaaaf688e1ebb7b5d54bfc938f\": container with ID starting with 413db7d5fa1756bfe36d375d27b2c80d9d0568eaaaf688e1ebb7b5d54bfc938f not found: ID does not exist" containerID="413db7d5fa1756bfe36d375d27b2c80d9d0568eaaaf688e1ebb7b5d54bfc938f"
Sep 09 06:08:08.828350 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.828279    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413db7d5fa1756bfe36d375d27b2c80d9d0568eaaaf688e1ebb7b5d54bfc938f"} err="failed to get container status \"413db7d5fa1756bfe36d375d27b2c80d9d0568eaaaf688e1ebb7b5d54bfc938f\": rpc error: code = NotFound desc = could not find container \"413db7d5fa1756bfe36d375d27b2c80d9d0568eaaaf688e1ebb7b5d54bfc938f\": container with ID starting with 413db7d5fa1756bfe36d375d27b2c80d9d0568eaaaf688e1ebb7b5d54bfc938f not found: ID does not exist"
Sep 09 06:08:08.842455 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.842403    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"]
Sep 09 06:08:08.847471 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:08.847442    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ecd13e/test-model-ecd13e-kserve-6c64c5b887-px62z"]
Sep 09 06:08:09.309736 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:09.309688    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da978ab-db7e-44ea-893e-cd60c07bae07" path="/var/lib/kubelet/pods/0da978ab-db7e-44ea-893e-cd60c07bae07/volumes"
Sep 09 06:08:11.865077 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:11.865044    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"]
Sep 09 06:08:11.865525 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:11.865237    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5" podUID="6263b891-ad1a-4c05-9754-5c75cb38b50c" containerName="maas-api" containerID="cri-o://8f7636291c6a3d4ec11f95cbae14aab1e9708f1b9f547ebec762ac687c29e4b5" gracePeriod=30
Sep 09 06:08:12.149613 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.149583    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:08:12.255747 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.255708    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll4dj\" (UniqueName: \"kubernetes.io/projected/6263b891-ad1a-4c05-9754-5c75cb38b50c-kube-api-access-ll4dj\") pod \"6263b891-ad1a-4c05-9754-5c75cb38b50c\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") "
Sep 09 06:08:12.255936 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.255785    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6263b891-ad1a-4c05-9754-5c75cb38b50c-metrics-certs\") pod \"6263b891-ad1a-4c05-9754-5c75cb38b50c\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") "
Sep 09 06:08:12.255936 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.255896    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6263b891-ad1a-4c05-9754-5c75cb38b50c-openshift-service-ca\") pod \"6263b891-ad1a-4c05-9754-5c75cb38b50c\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") "
Sep 09 06:08:12.256075 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.256042    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6263b891-ad1a-4c05-9754-5c75cb38b50c-maas-api-tls\") pod \"6263b891-ad1a-4c05-9754-5c75cb38b50c\" (UID: \"6263b891-ad1a-4c05-9754-5c75cb38b50c\") "
Sep 09 06:08:12.256276 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.256241    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6263b891-ad1a-4c05-9754-5c75cb38b50c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6263b891-ad1a-4c05-9754-5c75cb38b50c" (UID: "6263b891-ad1a-4c05-9754-5c75cb38b50c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:08:12.256469 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.256447    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6263b891-ad1a-4c05-9754-5c75cb38b50c-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:12.257992 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.257968    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6263b891-ad1a-4c05-9754-5c75cb38b50c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6263b891-ad1a-4c05-9754-5c75cb38b50c" (UID: "6263b891-ad1a-4c05-9754-5c75cb38b50c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:08:12.258152 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.258128    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6263b891-ad1a-4c05-9754-5c75cb38b50c-kube-api-access-ll4dj" (OuterVolumeSpecName: "kube-api-access-ll4dj") pod "6263b891-ad1a-4c05-9754-5c75cb38b50c" (UID: "6263b891-ad1a-4c05-9754-5c75cb38b50c"). InnerVolumeSpecName "kube-api-access-ll4dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:08:12.258383 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.258359    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6263b891-ad1a-4c05-9754-5c75cb38b50c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6263b891-ad1a-4c05-9754-5c75cb38b50c" (UID: "6263b891-ad1a-4c05-9754-5c75cb38b50c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:08:12.357613 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.357568    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6263b891-ad1a-4c05-9754-5c75cb38b50c-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:12.357613 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.357604    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll4dj\" (UniqueName: \"kubernetes.io/projected/6263b891-ad1a-4c05-9754-5c75cb38b50c-kube-api-access-ll4dj\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:12.357613 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.357619    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6263b891-ad1a-4c05-9754-5c75cb38b50c-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:12.676692 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.676653    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 06:08:12.676868 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.676730    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj" podUID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 06:08:12.848060 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.848013    2564 generic.go:356] "Generic (PLEG): container finished" podID="6263b891-ad1a-4c05-9754-5c75cb38b50c" containerID="8f7636291c6a3d4ec11f95cbae14aab1e9708f1b9f547ebec762ac687c29e4b5" exitCode=0
Sep 09 06:08:12.848060 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.848062    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"
Sep 09 06:08:12.848286 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.848054    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5" event={"ID":"6263b891-ad1a-4c05-9754-5c75cb38b50c","Type":"ContainerDied","Data":"8f7636291c6a3d4ec11f95cbae14aab1e9708f1b9f547ebec762ac687c29e4b5"}
Sep 09 06:08:12.848286 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.848167    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5" event={"ID":"6263b891-ad1a-4c05-9754-5c75cb38b50c","Type":"ContainerDied","Data":"7b94ed9afe1ecfe32626400d789df528380e1313620f96046fb2abcc13402f3b"}
Sep 09 06:08:12.848286 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.848195    2564 scope.go:117] "RemoveContainer" containerID="8f7636291c6a3d4ec11f95cbae14aab1e9708f1b9f547ebec762ac687c29e4b5"
Sep 09 06:08:12.858481 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.858463    2564 scope.go:117] "RemoveContainer" containerID="8f7636291c6a3d4ec11f95cbae14aab1e9708f1b9f547ebec762ac687c29e4b5"
Sep 09 06:08:12.858808 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:08:12.858789    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7636291c6a3d4ec11f95cbae14aab1e9708f1b9f547ebec762ac687c29e4b5\": container with ID starting with 8f7636291c6a3d4ec11f95cbae14aab1e9708f1b9f547ebec762ac687c29e4b5 not found: ID does not exist" containerID="8f7636291c6a3d4ec11f95cbae14aab1e9708f1b9f547ebec762ac687c29e4b5"
Sep 09 06:08:12.858875 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.858820    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7636291c6a3d4ec11f95cbae14aab1e9708f1b9f547ebec762ac687c29e4b5"} err="failed to get container status \"8f7636291c6a3d4ec11f95cbae14aab1e9708f1b9f547ebec762ac687c29e4b5\": rpc error: code = NotFound desc = could not find container \"8f7636291c6a3d4ec11f95cbae14aab1e9708f1b9f547ebec762ac687c29e4b5\": container with ID starting with 8f7636291c6a3d4ec11f95cbae14aab1e9708f1b9f547ebec762ac687c29e4b5 not found: ID does not exist"
Sep 09 06:08:12.873130 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.873101    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"]
Sep 09 06:08:12.877544 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:12.877522    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd13e-7db6db4c8f-r4hz5"]
Sep 09 06:08:13.303647 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:13.303613    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6263b891-ad1a-4c05-9754-5c75cb38b50c" path="/var/lib/kubelet/pods/6263b891-ad1a-4c05-9754-5c75cb38b50c/volumes"
Sep 09 06:08:14.862895 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:14.862864    2564 generic.go:356] "Generic (PLEG): container finished" podID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerID="11cef41c04716d70dd62e77976c5adf97318fd77e1db4ea87bc72b1b3c308e06" exitCode=0
Sep 09 06:08:14.863310 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:14.862936    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj" event={"ID":"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f","Type":"ContainerDied","Data":"11cef41c04716d70dd62e77976c5adf97318fd77e1db4ea87bc72b1b3c308e06"}
Sep 09 06:08:14.863383 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:14.863367    2564 scope.go:117] "RemoveContainer" containerID="11cef41c04716d70dd62e77976c5adf97318fd77e1db4ea87bc72b1b3c308e06"
Sep 09 06:08:14.864578 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:14.864550    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:08:14.864642 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:14.864615    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:08:14.864683 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:14.864646    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:08:15.873432 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:15.873397    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj" event={"ID":"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f","Type":"ContainerStarted","Data":"3d7d2d9aa72a8dab48aaddf5539209ef6b37a983086c78db9bd9f3a9e9f377ff"}
Sep 09 06:08:16.074704 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.074660    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh" podUID="1182f86e-73cd-48a6-8ac3-f4f9a80af43e" containerName="main" containerID="cri-o://a192b6094b38a3ad93aa8547b998e0cf7f943269d293184d524a4aece9ef0e17" gracePeriod=45
Sep 09 06:08:16.375140 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.375113    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:08:16.501162 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.501077    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-dshm\") pod \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") "
Sep 09 06:08:16.501162 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.501120    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42tcj\" (UniqueName: \"kubernetes.io/projected/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-kube-api-access-42tcj\") pod \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") "
Sep 09 06:08:16.501162 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.501145    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-tmp-dir\") pod \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") "
Sep 09 06:08:16.501450 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.501173    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-tls-certs\") pod \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") "
Sep 09 06:08:16.501450 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.501208    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-model-cache\") pod \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") "
Sep 09 06:08:16.501450 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.501267    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-home\") pod \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\" (UID: \"1182f86e-73cd-48a6-8ac3-f4f9a80af43e\") "
Sep 09 06:08:16.501450 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.501410    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1182f86e-73cd-48a6-8ac3-f4f9a80af43e" (UID: "1182f86e-73cd-48a6-8ac3-f4f9a80af43e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:08:16.501680 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.501449    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-model-cache" (OuterVolumeSpecName: "model-cache") pod "1182f86e-73cd-48a6-8ac3-f4f9a80af43e" (UID: "1182f86e-73cd-48a6-8ac3-f4f9a80af43e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:08:16.501680 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.501550    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-home" (OuterVolumeSpecName: "home") pod "1182f86e-73cd-48a6-8ac3-f4f9a80af43e" (UID: "1182f86e-73cd-48a6-8ac3-f4f9a80af43e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:08:16.501753 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.501718    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-home\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:16.501753 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.501740    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-tmp-dir\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:16.501832 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.501756    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-model-cache\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:16.503462 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.503428    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-dshm" (OuterVolumeSpecName: "dshm") pod "1182f86e-73cd-48a6-8ac3-f4f9a80af43e" (UID: "1182f86e-73cd-48a6-8ac3-f4f9a80af43e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:08:16.503603 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.503458    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1182f86e-73cd-48a6-8ac3-f4f9a80af43e" (UID: "1182f86e-73cd-48a6-8ac3-f4f9a80af43e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:08:16.503603 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.503557    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-kube-api-access-42tcj" (OuterVolumeSpecName: "kube-api-access-42tcj") pod "1182f86e-73cd-48a6-8ac3-f4f9a80af43e" (UID: "1182f86e-73cd-48a6-8ac3-f4f9a80af43e"). InnerVolumeSpecName "kube-api-access-42tcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:08:16.602836 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.602801    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-dshm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:16.602836 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.602829    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42tcj\" (UniqueName: \"kubernetes.io/projected/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-kube-api-access-42tcj\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:16.602836 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.602839    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1182f86e-73cd-48a6-8ac3-f4f9a80af43e-tls-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:08:16.658827 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.658786    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:08:16.659269 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.659243    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 09 06:08:16.659352 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.659320    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj" podUID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 09 06:08:16.883981 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.883947    2564 generic.go:356] "Generic (PLEG): container finished" podID="1182f86e-73cd-48a6-8ac3-f4f9a80af43e" containerID="a192b6094b38a3ad93aa8547b998e0cf7f943269d293184d524a4aece9ef0e17" exitCode=0
Sep 09 06:08:16.884401 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.883989    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"
Sep 09 06:08:16.884401 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.884033    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh" event={"ID":"1182f86e-73cd-48a6-8ac3-f4f9a80af43e","Type":"ContainerDied","Data":"a192b6094b38a3ad93aa8547b998e0cf7f943269d293184d524a4aece9ef0e17"}
Sep 09 06:08:16.884401 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.884077    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh" event={"ID":"1182f86e-73cd-48a6-8ac3-f4f9a80af43e","Type":"ContainerDied","Data":"fb26e16da5ad8ef4faa52abc7a5fc0e46ffd1afaf8fdc6cfd10fc880fb84e62d"}
Sep 09 06:08:16.884401 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.884095    2564 scope.go:117] "RemoveContainer" containerID="a192b6094b38a3ad93aa8547b998e0cf7f943269d293184d524a4aece9ef0e17"
Sep 09 06:08:16.893898 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.893881    2564 scope.go:117] "RemoveContainer" containerID="a192b6094b38a3ad93aa8547b998e0cf7f943269d293184d524a4aece9ef0e17"
Sep 09 06:08:16.894138 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:08:16.894123    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a192b6094b38a3ad93aa8547b998e0cf7f943269d293184d524a4aece9ef0e17\": container with ID starting with a192b6094b38a3ad93aa8547b998e0cf7f943269d293184d524a4aece9ef0e17 not found: ID does not exist" containerID="a192b6094b38a3ad93aa8547b998e0cf7f943269d293184d524a4aece9ef0e17"
Sep 09 06:08:16.894188 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.894144    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a192b6094b38a3ad93aa8547b998e0cf7f943269d293184d524a4aece9ef0e17"} err="failed to get container status \"a192b6094b38a3ad93aa8547b998e0cf7f943269d293184d524a4aece9ef0e17\": rpc error: code = NotFound desc = could not find container \"a192b6094b38a3ad93aa8547b998e0cf7f943269d293184d524a4aece9ef0e17\": container with ID starting with a192b6094b38a3ad93aa8547b998e0cf7f943269d293184d524a4aece9ef0e17 not found: ID does not exist"
Sep 09 06:08:16.905041 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.905016    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"]
Sep 09 06:08:16.909261 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:16.909240    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fe8eaa/test-model-fe8eaa-kserve-6f5d8d6c7c-r77qh"]
Sep 09 06:08:17.303422 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:17.303343    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1182f86e-73cd-48a6-8ac3-f4f9a80af43e" path="/var/lib/kubelet/pods/1182f86e-73cd-48a6-8ac3-f4f9a80af43e/volumes"
Sep 09 06:08:17.659555 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:17.659483    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 09 06:08:17.659755 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:17.659582    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj" podUID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 09 06:08:18.238126 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.236831    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"]
Sep 09 06:08:18.239436 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.239402    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6263b891-ad1a-4c05-9754-5c75cb38b50c" containerName="maas-api"
Sep 09 06:08:18.239436 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.239432    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6263b891-ad1a-4c05-9754-5c75cb38b50c" containerName="maas-api"
Sep 09 06:08:18.239723 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.239466    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da978ab-db7e-44ea-893e-cd60c07bae07" containerName="main"
Sep 09 06:08:18.239723 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.239478    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da978ab-db7e-44ea-893e-cd60c07bae07" containerName="main"
Sep 09 06:08:18.239723 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.239587    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1182f86e-73cd-48a6-8ac3-f4f9a80af43e" containerName="main"
Sep 09 06:08:18.239723 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.239617    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1182f86e-73cd-48a6-8ac3-f4f9a80af43e" containerName="main"
Sep 09 06:08:18.255981 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.248775    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1182f86e-73cd-48a6-8ac3-f4f9a80af43e" containerName="main"
Sep 09 06:08:18.255981 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.248807    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6263b891-ad1a-4c05-9754-5c75cb38b50c" containerName="maas-api"
Sep 09 06:08:18.255981 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.248824    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da978ab-db7e-44ea-893e-cd60c07bae07" containerName="main"
Sep 09 06:08:18.255981 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.254529    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.276727 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.276695    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-43eb2d\"/\"unconfigured-w4-43eb2d-kserve-self-signed-certs\""
Sep 09 06:08:18.277700 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.277004    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-43eb2d\"/\"default-dockercfg-j4z2r\""
Sep 09 06:08:18.278027 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.277125    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-43eb2d\"/\"kube-root-ca.crt\""
Sep 09 06:08:18.278246 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.277174    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-43eb2d\"/\"openshift-service-ca.crt\""
Sep 09 06:08:18.280335 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.280259    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"]
Sep 09 06:08:18.321282 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.321250    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5r2z\" (UniqueName: \"kubernetes.io/projected/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-kube-api-access-h5r2z\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.321428 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.321294    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-tmp-dir\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.321428 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.321351    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-tls-certs\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.321428 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.321397    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-home\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.321428 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.321415    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-model-cache\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.321599 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.321519    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-dshm\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.422433 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.422385    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-tls-certs\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.422631 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.422456    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-home\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.422631 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.422478    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-model-cache\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.422631 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.422573    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-dshm\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.422802 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.422658    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5r2z\" (UniqueName: \"kubernetes.io/projected/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-kube-api-access-h5r2z\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.422802 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.422710    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-tmp-dir\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.422929 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.422905    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-home\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.422992 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.422959    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-model-cache\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.423156 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.423138    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-tmp-dir\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.424910 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.424890    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-dshm\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.425339 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.425323    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-tls-certs\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.433656 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.433633    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5r2z\" (UniqueName: \"kubernetes.io/projected/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-kube-api-access-h5r2z\") pod \"unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") " pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.584795 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.584763    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:18.672967 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.672884    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 09 06:08:18.673115 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.672972    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj" podUID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 09 06:08:18.871853 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.870657    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"]
Sep 09 06:08:18.872677 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:08:18.872603    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod132aa2fd_ce5b_4146_9d07_ebacfa2663e5.slice/crio-ab9e261648d41f5dc134cc47666e774e8adc60944757042f74ca28f8083ee34a WatchSource:0}: Error finding container ab9e261648d41f5dc134cc47666e774e8adc60944757042f74ca28f8083ee34a: Status 404 returned error can't find the container with id ab9e261648d41f5dc134cc47666e774e8adc60944757042f74ca28f8083ee34a
Sep 09 06:08:18.898834 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:18.898800    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp" event={"ID":"132aa2fd-ce5b-4146-9d07-ebacfa2663e5","Type":"ContainerStarted","Data":"ab9e261648d41f5dc134cc47666e774e8adc60944757042f74ca28f8083ee34a"}
Sep 09 06:08:19.658635 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:19.658593    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 09 06:08:19.659137 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:19.658660    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj" podUID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 09 06:08:19.907408 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:19.907373    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp" event={"ID":"132aa2fd-ce5b-4146-9d07-ebacfa2663e5","Type":"ContainerStarted","Data":"c54db14705d04b1d6c28e6d327b89b37a3d8dba32aaa8da6e86e8e33ed91fa65"}
Sep 09 06:08:19.925438 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:19.925262    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp" podStartSLOduration=1.925240493 podStartE2EDuration="1.925240493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:08:19.922419741 +0000 UTC m=+1127.138749677" watchObservedRunningTime="2026-09-09 06:08:19.925240493 +0000 UTC m=+1127.141570380"
Sep 09 06:08:20.659558 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:20.659514    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 09 06:08:20.660038 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:20.659582    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj" podUID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 09 06:08:20.989740 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:20.989647    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"]
Sep 09 06:08:20.994671 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:20.994632    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:20.997613 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:20.997576    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-389cd1da-openshift-default-dockercfg-b4hqg\""
Sep 09 06:08:21.000735 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.000708    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"]
Sep 09 06:08:21.055605 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.055566    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-credential-socket\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.055778 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.055636    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-envoy\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.055778 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.055663    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-data\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.055778 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.055712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-podinfo\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.055778 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.055731    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-workload-certs\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.055972 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.055813    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-token\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.055972 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.055844    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adf1feb-cf04-4b30-8c52-bdd520849eab-istiod-ca-cert\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.055972 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.055865    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-workload-socket\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.055972 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.055881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x95z8\" (UniqueName: \"kubernetes.io/projected/7adf1feb-cf04-4b30-8c52-bdd520849eab-kube-api-access-x95z8\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.156740 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.156698    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-credential-socket\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.156942 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.156758    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-envoy\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.156942 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.156783    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-data\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.156942 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.156835    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-podinfo\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.156942 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.156863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-workload-certs\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.156942 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.156921    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-token\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.157225 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.156957    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adf1feb-cf04-4b30-8c52-bdd520849eab-istiod-ca-cert\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.157225 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.156988    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-workload-socket\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.157225 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.157011    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x95z8\" (UniqueName: \"kubernetes.io/projected/7adf1feb-cf04-4b30-8c52-bdd520849eab-kube-api-access-x95z8\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.157225 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.157178    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-credential-socket\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.157755 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.157719    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-workload-socket\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.157755 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.157741    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-data\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.157901 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.157717    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adf1feb-cf04-4b30-8c52-bdd520849eab-istiod-ca-cert\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.158008 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.157985    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-workload-certs\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.159237 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.159220    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-envoy\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.159664 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.159641    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-podinfo\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.165627 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.165598    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x95z8\" (UniqueName: \"kubernetes.io/projected/7adf1feb-cf04-4b30-8c52-bdd520849eab-kube-api-access-x95z8\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.165735 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.165715    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-token\") pod \"e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") " pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.307785 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.307703    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:21.587679 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.587651    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"]
Sep 09 06:08:21.588594 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:08:21.588560    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adf1feb_cf04_4b30_8c52_bdd520849eab.slice/crio-082b5ec06d015a6c9e934804529d16a4cb1b317f38fbace8a21a788d2bf28dae WatchSource:0}: Error finding container 082b5ec06d015a6c9e934804529d16a4cb1b317f38fbace8a21a788d2bf28dae: Status 404 returned error can't find the container with id 082b5ec06d015a6c9e934804529d16a4cb1b317f38fbace8a21a788d2bf28dae
Sep 09 06:08:21.591093 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.591061    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:08:21.591174 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.591130    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:08:21.591214 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.591185    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:08:21.663311 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.663285    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:08:21.663774 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.663748    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:08:21.664528 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.664510    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:08:21.923709 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.923623    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs" event={"ID":"7adf1feb-cf04-4b30-8c52-bdd520849eab","Type":"ContainerStarted","Data":"9900f745c1117a1298316f6dfa5abbbc5c85f5a54e64f019d80e07a25533205a"}
Sep 09 06:08:21.923709 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.923680    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs" event={"ID":"7adf1feb-cf04-4b30-8c52-bdd520849eab","Type":"ContainerStarted","Data":"082b5ec06d015a6c9e934804529d16a4cb1b317f38fbace8a21a788d2bf28dae"}
Sep 09 06:08:21.945088 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:21.945035    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs" podStartSLOduration=1.945020623 podStartE2EDuration="1.945020623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:08:21.94164758 +0000 UTC m=+1129.157977469" watchObservedRunningTime="2026-09-09 06:08:21.945020623 +0000 UTC m=+1129.161350506"
Sep 09 06:08:22.308617 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:22.308525    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:22.314207 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:22.314182    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:22.930106 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:22.930057    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:22.931099 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:22.931080    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:28.585605 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:28.585564    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:28.586027 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:28.585640    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:28.598413 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:28.598386    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:28.982138 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:28.982056    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:08:33.432571 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:33.432535    2564 scope.go:117] "RemoveContainer" containerID="2c3779946ac15eda8934b418f233de11da35c257b669640afefd1ed617925b0b"
Sep 09 06:08:44.122045 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.122013    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"]
Sep 09 06:08:44.125831 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.125809    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:44.128050 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.128028    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-6b5d89-serving-cert\""
Sep 09 06:08:44.135165 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.135139    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"]
Sep 09 06:08:44.211039 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.210997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:44.211039 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.211044    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr2m4\" (UniqueName: \"kubernetes.io/projected/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-kube-api-access-gr2m4\") pod \"maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:44.211238 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.211189    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:44.211238 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.211217    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:44.312682 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.312641    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:44.312883 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.312689    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:44.312883 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.312788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:44.312883 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.312815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr2m4\" (UniqueName: \"kubernetes.io/projected/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-kube-api-access-gr2m4\") pod \"maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:44.313058 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:08:44.312982    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-serving-cert: secret "maas-api-e2e-shared-a-w3-6b5d89-serving-cert" not found
Sep 09 06:08:44.313111 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:08:44.313084    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-maas-api-tls podName:d5e9cb12-8d68-45ea-92ed-246f90da8e6b nodeName:}" failed. No retries permitted until 2026-09-09 06:08:44.813057458 +0000 UTC m=+1152.029387338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-maas-api-tls") pod "maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl" (UID: "d5e9cb12-8d68-45ea-92ed-246f90da8e6b") : secret "maas-api-e2e-shared-a-w3-6b5d89-serving-cert" not found
Sep 09 06:08:44.313463 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.313438    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:44.315303 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.315279    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:44.323664 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.323637    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr2m4\" (UniqueName: \"kubernetes.io/projected/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-kube-api-access-gr2m4\") pod \"maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:44.817159 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.817119    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:44.819837 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:44.819803    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:45.040957 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:45.040655    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:45.329856 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:45.329828    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"]
Sep 09 06:08:45.331072 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:08:45.331035    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e9cb12_8d68_45ea_92ed_246f90da8e6b.slice/crio-ccb83c59ff5b4d92297fa19637ad1496eaacc08f9ea60323bba35c827714c3a7 WatchSource:0}: Error finding container ccb83c59ff5b4d92297fa19637ad1496eaacc08f9ea60323bba35c827714c3a7: Status 404 returned error can't find the container with id ccb83c59ff5b4d92297fa19637ad1496eaacc08f9ea60323bba35c827714c3a7
Sep 09 06:08:46.092287 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:46.092249    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl" event={"ID":"d5e9cb12-8d68-45ea-92ed-246f90da8e6b","Type":"ContainerStarted","Data":"c4a88e911ea81708b817677aafc40d30df8916b9d6842f4556c1703d4bf4ca18"}
Sep 09 06:08:46.092466 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:46.092312    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl" event={"ID":"d5e9cb12-8d68-45ea-92ed-246f90da8e6b","Type":"ContainerStarted","Data":"ccb83c59ff5b4d92297fa19637ad1496eaacc08f9ea60323bba35c827714c3a7"}
Sep 09 06:08:46.092466 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:46.092344    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:46.110047 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:46.109999    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl" podStartSLOduration=1.778130695 podStartE2EDuration="2.109968636s" totalImagesPullingTime="331.837941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:08:46.107765261 +0000 UTC m=+1153.324095128" watchObservedRunningTime="2026-09-09 06:08:46.109968636 +0000 UTC m=+1153.326298519"
Sep 09 06:08:47.099753 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:47.099720    2564 generic.go:356] "Generic (PLEG): container finished" podID="7adf1feb-cf04-4b30-8c52-bdd520849eab" containerID="9900f745c1117a1298316f6dfa5abbbc5c85f5a54e64f019d80e07a25533205a" exitCode=0
Sep 09 06:08:47.100149 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:47.099793    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs" event={"ID":"7adf1feb-cf04-4b30-8c52-bdd520849eab","Type":"ContainerDied","Data":"9900f745c1117a1298316f6dfa5abbbc5c85f5a54e64f019d80e07a25533205a"}
Sep 09 06:08:47.100269 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:47.100252    2564 scope.go:117] "RemoveContainer" containerID="9900f745c1117a1298316f6dfa5abbbc5c85f5a54e64f019d80e07a25533205a"
Sep 09 06:08:47.101370 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:47.101342    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:08:47.101530 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:47.101413    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:08:47.101530 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:47.101447    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 06:08:48.108969 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:48.108934    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs" event={"ID":"7adf1feb-cf04-4b30-8c52-bdd520849eab","Type":"ContainerStarted","Data":"02264ca99fa119720c38585e13f90a31b44a040a14d1752b4026830980212469"}
Sep 09 06:08:48.308153 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:48.308116    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:48.309308 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:48.309282    2564 patch_prober.go:28] interesting pod/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 09 06:08:48.309425 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:48.309335    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs" podUID="7adf1feb-cf04-4b30-8c52-bdd520849eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 09 06:08:49.308137 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:49.308103    2564 patch_prober.go:28] interesting pod/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 09 06:08:49.308555 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:49.308154    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs" podUID="7adf1feb-cf04-4b30-8c52-bdd520849eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 09 06:08:50.308299 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:50.308259    2564 patch_prober.go:28] interesting pod/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 09 06:08:50.308719 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:50.308321    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs" podUID="7adf1feb-cf04-4b30-8c52-bdd520849eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 09 06:08:51.309050 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:51.309015    2564 patch_prober.go:28] interesting pod/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 09 06:08:51.309443 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:51.309068    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs" podUID="7adf1feb-cf04-4b30-8c52-bdd520849eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 09 06:08:52.105451 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:52.105420    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:08:52.308242 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:52.308206    2564 patch_prober.go:28] interesting pod/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 09 06:08:52.308416 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:52.308262    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs" podUID="7adf1feb-cf04-4b30-8c52-bdd520849eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 09 06:08:52.930305 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:52.930270    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:53.312374 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:53.312303    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:08:53.313272 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:08:53.313250    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:09:12.693844 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.693811    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"]
Sep 09 06:09:12.697665 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.697642    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:12.699999 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.699975    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-7cf6be-serving-cert\""
Sep 09 06:09:12.706413 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.706380    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"]
Sep 09 06:09:12.811820 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.811776    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5abff106-d69b-418e-a836-cb923ec4cee0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:12.812016 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.811920    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5abff106-d69b-418e-a836-cb923ec4cee0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:12.812016 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.811953    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsp9b\" (UniqueName: \"kubernetes.io/projected/5abff106-d69b-418e-a836-cb923ec4cee0-kube-api-access-zsp9b\") pod \"maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:12.812146 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.812021    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abff106-d69b-418e-a836-cb923ec4cee0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:12.912594 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.912558    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5abff106-d69b-418e-a836-cb923ec4cee0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:12.912775 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.912601    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsp9b\" (UniqueName: \"kubernetes.io/projected/5abff106-d69b-418e-a836-cb923ec4cee0-kube-api-access-zsp9b\") pod \"maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:12.912775 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.912664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abff106-d69b-418e-a836-cb923ec4cee0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:12.912775 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.912736    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5abff106-d69b-418e-a836-cb923ec4cee0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:12.913388 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.913362    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abff106-d69b-418e-a836-cb923ec4cee0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:12.915261 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.915229    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5abff106-d69b-418e-a836-cb923ec4cee0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:12.915349 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.915229    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5abff106-d69b-418e-a836-cb923ec4cee0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:12.920127 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:12.920106    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsp9b\" (UniqueName: \"kubernetes.io/projected/5abff106-d69b-418e-a836-cb923ec4cee0-kube-api-access-zsp9b\") pod \"maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:13.011379 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:13.011292    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:13.294024 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:13.293874    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"]
Sep 09 06:09:13.295368 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:09:13.295326    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abff106_d69b_418e_a836_cb923ec4cee0.slice/crio-a035f750c4e1f5b85010a925e192c1f6dc555b4dcb37943ae66fb63c910d07e0 WatchSource:0}: Error finding container a035f750c4e1f5b85010a925e192c1f6dc555b4dcb37943ae66fb63c910d07e0: Status 404 returned error can't find the container with id a035f750c4e1f5b85010a925e192c1f6dc555b4dcb37943ae66fb63c910d07e0
Sep 09 06:09:14.281827 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:14.281789    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7" event={"ID":"5abff106-d69b-418e-a836-cb923ec4cee0","Type":"ContainerStarted","Data":"3abaabebcd2a9f4c828a494fbda56080b830d5c5e8c1d9b3a3c5933555d1de61"}
Sep 09 06:09:14.282294 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:14.281843    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7" event={"ID":"5abff106-d69b-418e-a836-cb923ec4cee0","Type":"ContainerStarted","Data":"a035f750c4e1f5b85010a925e192c1f6dc555b4dcb37943ae66fb63c910d07e0"}
Sep 09 06:09:14.282294 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:14.281927    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:14.298736 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:14.298692    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7" podStartSLOduration=1.8698067520000001 podStartE2EDuration="2.298678995s" totalImagesPullingTime="428.872243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:09:14.29768135 +0000 UTC m=+1181.514011234" watchObservedRunningTime="2026-09-09 06:09:14.298678995 +0000 UTC m=+1181.515008907"
Sep 09 06:09:20.293555 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:20.293519    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:09:33.362266 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:33.362238    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 06:09:33.368707 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:33.368686    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 06:09:33.468926 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:09:33.468890    2564 scope.go:117] "RemoveContainer" containerID="2f3556c19175a64fd979e7a2d65dded33e9da503fe586355d6dfaadfc1c9498c"
Sep 09 06:10:33.494828 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:10:33.494760    2564 scope.go:117] "RemoveContainer" containerID="7e2c239c9efb0518dfeaf97dc5a06f2d66d01c646dc7102d29e9846d46771138"
Sep 09 06:11:04.944310 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:04.944274    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"]
Sep 09 06:11:04.944820 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:04.944473    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs" podUID="7adf1feb-cf04-4b30-8c52-bdd520849eab" containerName="istio-proxy" containerID="cri-o://02264ca99fa119720c38585e13f90a31b44a040a14d1752b4026830980212469" gracePeriod=30
Sep 09 06:11:08.315644 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:08.315609    2564 patch_prober.go:28] interesting pod/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 06:11:08.316002 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:08.315683    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs" podUID="7adf1feb-cf04-4b30-8c52-bdd520849eab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 06:11:10.201532 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.201508    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:11:10.325932 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.325898    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-workload-certs\") pod \"7adf1feb-cf04-4b30-8c52-bdd520849eab\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") "
Sep 09 06:11:10.326126 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.325957    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-data\") pod \"7adf1feb-cf04-4b30-8c52-bdd520849eab\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") "
Sep 09 06:11:10.326126 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326010    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-token\") pod \"7adf1feb-cf04-4b30-8c52-bdd520849eab\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") "
Sep 09 06:11:10.326126 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326085    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x95z8\" (UniqueName: \"kubernetes.io/projected/7adf1feb-cf04-4b30-8c52-bdd520849eab-kube-api-access-x95z8\") pod \"7adf1feb-cf04-4b30-8c52-bdd520849eab\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") "
Sep 09 06:11:10.326296 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326160    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adf1feb-cf04-4b30-8c52-bdd520849eab-istiod-ca-cert\") pod \"7adf1feb-cf04-4b30-8c52-bdd520849eab\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") "
Sep 09 06:11:10.326296 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326199    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-podinfo\") pod \"7adf1feb-cf04-4b30-8c52-bdd520849eab\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") "
Sep 09 06:11:10.326296 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326197    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7adf1feb-cf04-4b30-8c52-bdd520849eab" (UID: "7adf1feb-cf04-4b30-8c52-bdd520849eab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:11:10.326296 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326244    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-workload-socket\") pod \"7adf1feb-cf04-4b30-8c52-bdd520849eab\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") "
Sep 09 06:11:10.326296 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326256    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-data" (OuterVolumeSpecName: "istio-data") pod "7adf1feb-cf04-4b30-8c52-bdd520849eab" (UID: "7adf1feb-cf04-4b30-8c52-bdd520849eab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:11:10.326578 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326297    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-credential-socket\") pod \"7adf1feb-cf04-4b30-8c52-bdd520849eab\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") "
Sep 09 06:11:10.326578 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326330    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-envoy\") pod \"7adf1feb-cf04-4b30-8c52-bdd520849eab\" (UID: \"7adf1feb-cf04-4b30-8c52-bdd520849eab\") "
Sep 09 06:11:10.326666 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326558    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7adf1feb-cf04-4b30-8c52-bdd520849eab" (UID: "7adf1feb-cf04-4b30-8c52-bdd520849eab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:11:10.326666 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326638    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7adf1feb-cf04-4b30-8c52-bdd520849eab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7adf1feb-cf04-4b30-8c52-bdd520849eab" (UID: "7adf1feb-cf04-4b30-8c52-bdd520849eab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:11:10.326778 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326741    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adf1feb-cf04-4b30-8c52-bdd520849eab-istiod-ca-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:11:10.326778 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326761    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-workload-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:11:10.326778 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326774    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-workload-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:11:10.326944 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326787    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-data\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:11:10.326944 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.326873    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7adf1feb-cf04-4b30-8c52-bdd520849eab" (UID: "7adf1feb-cf04-4b30-8c52-bdd520849eab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:11:10.328371 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.328342    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adf1feb-cf04-4b30-8c52-bdd520849eab-kube-api-access-x95z8" (OuterVolumeSpecName: "kube-api-access-x95z8") pod "7adf1feb-cf04-4b30-8c52-bdd520849eab" (UID: "7adf1feb-cf04-4b30-8c52-bdd520849eab"). InnerVolumeSpecName "kube-api-access-x95z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:11:10.328518 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.328382    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-token" (OuterVolumeSpecName: "istio-token") pod "7adf1feb-cf04-4b30-8c52-bdd520849eab" (UID: "7adf1feb-cf04-4b30-8c52-bdd520849eab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:11:10.328518 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.328402    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7adf1feb-cf04-4b30-8c52-bdd520849eab" (UID: "7adf1feb-cf04-4b30-8c52-bdd520849eab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 06:11:10.328611 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.328481    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7adf1feb-cf04-4b30-8c52-bdd520849eab" (UID: "7adf1feb-cf04-4b30-8c52-bdd520849eab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:11:10.427822 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.427774    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x95z8\" (UniqueName: \"kubernetes.io/projected/7adf1feb-cf04-4b30-8c52-bdd520849eab-kube-api-access-x95z8\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:11:10.427822 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.427821    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-podinfo\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:11:10.428003 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.427836    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-credential-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:11:10.428003 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.427849    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-envoy\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:11:10.428003 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:10.427862    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adf1feb-cf04-4b30-8c52-bdd520849eab-istio-token\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:11:11.015351 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:11.015260    2564 generic.go:356] "Generic (PLEG): container finished" podID="7adf1feb-cf04-4b30-8c52-bdd520849eab" containerID="02264ca99fa119720c38585e13f90a31b44a040a14d1752b4026830980212469" exitCode=0
Sep 09 06:11:11.015351 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:11.015302    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs" event={"ID":"7adf1feb-cf04-4b30-8c52-bdd520849eab","Type":"ContainerDied","Data":"02264ca99fa119720c38585e13f90a31b44a040a14d1752b4026830980212469"}
Sep 09 06:11:11.015351 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:11.015312    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"
Sep 09 06:11:11.015351 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:11.015344    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs" event={"ID":"7adf1feb-cf04-4b30-8c52-bdd520849eab","Type":"ContainerDied","Data":"082b5ec06d015a6c9e934804529d16a4cb1b317f38fbace8a21a788d2bf28dae"}
Sep 09 06:11:11.015720 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:11.015365    2564 scope.go:117] "RemoveContainer" containerID="02264ca99fa119720c38585e13f90a31b44a040a14d1752b4026830980212469"
Sep 09 06:11:11.028504 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:11.028476    2564 scope.go:117] "RemoveContainer" containerID="9900f745c1117a1298316f6dfa5abbbc5c85f5a54e64f019d80e07a25533205a"
Sep 09 06:11:11.037959 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:11.037937    2564 scope.go:117] "RemoveContainer" containerID="02264ca99fa119720c38585e13f90a31b44a040a14d1752b4026830980212469"
Sep 09 06:11:11.038252 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:11:11.038231    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02264ca99fa119720c38585e13f90a31b44a040a14d1752b4026830980212469\": container with ID starting with 02264ca99fa119720c38585e13f90a31b44a040a14d1752b4026830980212469 not found: ID does not exist" containerID="02264ca99fa119720c38585e13f90a31b44a040a14d1752b4026830980212469"
Sep 09 06:11:11.038340 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:11.038264    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02264ca99fa119720c38585e13f90a31b44a040a14d1752b4026830980212469"} err="failed to get container status \"02264ca99fa119720c38585e13f90a31b44a040a14d1752b4026830980212469\": rpc error: code = NotFound desc = could not find container \"02264ca99fa119720c38585e13f90a31b44a040a14d1752b4026830980212469\": container with ID starting with 02264ca99fa119720c38585e13f90a31b44a040a14d1752b4026830980212469 not found: ID does not exist"
Sep 09 06:11:11.038340 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:11.038290    2564 scope.go:117] "RemoveContainer" containerID="9900f745c1117a1298316f6dfa5abbbc5c85f5a54e64f019d80e07a25533205a"
Sep 09 06:11:11.038620 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:11:11.038595    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9900f745c1117a1298316f6dfa5abbbc5c85f5a54e64f019d80e07a25533205a\": container with ID starting with 9900f745c1117a1298316f6dfa5abbbc5c85f5a54e64f019d80e07a25533205a not found: ID does not exist" containerID="9900f745c1117a1298316f6dfa5abbbc5c85f5a54e64f019d80e07a25533205a"
Sep 09 06:11:11.038680 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:11.038630    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9900f745c1117a1298316f6dfa5abbbc5c85f5a54e64f019d80e07a25533205a"} err="failed to get container status \"9900f745c1117a1298316f6dfa5abbbc5c85f5a54e64f019d80e07a25533205a\": rpc error: code = NotFound desc = could not find container \"9900f745c1117a1298316f6dfa5abbbc5c85f5a54e64f019d80e07a25533205a\": container with ID starting with 9900f745c1117a1298316f6dfa5abbbc5c85f5a54e64f019d80e07a25533205a not found: ID does not exist"
Sep 09 06:11:11.041247 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:11.041219    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"]
Sep 09 06:11:11.044307 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:11.044287    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-389cd1da-openshift-default-866cdf9b48-w4wjs"]
Sep 09 06:11:11.303813 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:11:11.303738    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7adf1feb-cf04-4b30-8c52-bdd520849eab" path="/var/lib/kubelet/pods/7adf1feb-cf04-4b30-8c52-bdd520849eab/volumes"
Sep 09 06:12:03.305280 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.305173    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"]
Sep 09 06:12:03.305863 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.305719    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7adf1feb-cf04-4b30-8c52-bdd520849eab" containerName="istio-proxy"
Sep 09 06:12:03.305863 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.305733    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7adf1feb-cf04-4b30-8c52-bdd520849eab" containerName="istio-proxy"
Sep 09 06:12:03.305863 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.305754    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7adf1feb-cf04-4b30-8c52-bdd520849eab" containerName="istio-proxy"
Sep 09 06:12:03.305863 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.305760    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7adf1feb-cf04-4b30-8c52-bdd520849eab" containerName="istio-proxy"
Sep 09 06:12:03.305863 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.305838    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7adf1feb-cf04-4b30-8c52-bdd520849eab" containerName="istio-proxy"
Sep 09 06:12:03.305863 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.305851    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7adf1feb-cf04-4b30-8c52-bdd520849eab" containerName="istio-proxy"
Sep 09 06:12:03.309066 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.309050    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:03.311370 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.311345    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e1ed02-serving-cert\""
Sep 09 06:12:03.316590 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.316567    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"]
Sep 09 06:12:03.356510 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.356470    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqwzk\" (UniqueName: \"kubernetes.io/projected/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-kube-api-access-fqwzk\") pod \"maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:03.356688 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.356591    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-maas-api-tls\") pod \"maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:03.356688 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.356647    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-openshift-service-ca\") pod \"maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:03.356688 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.356682    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-metrics-certs\") pod \"maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:03.457391 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.457343    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-openshift-service-ca\") pod \"maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:03.457614 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.457401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-metrics-certs\") pod \"maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:03.457614 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.457507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqwzk\" (UniqueName: \"kubernetes.io/projected/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-kube-api-access-fqwzk\") pod \"maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:03.457614 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.457573    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-maas-api-tls\") pod \"maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:03.457797 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:12:03.457696    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-serving-cert: secret "maas-api-e2e-api-a-e1ed02-serving-cert" not found
Sep 09 06:12:03.457797 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:12:03.457779    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-maas-api-tls podName:7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3 nodeName:}" failed. No retries permitted until 2026-09-09 06:12:03.957756093 +0000 UTC m=+1351.174085954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-maas-api-tls") pod "maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz" (UID: "7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3") : secret "maas-api-e2e-api-a-e1ed02-serving-cert" not found
Sep 09 06:12:03.458237 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.458210    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-openshift-service-ca\") pod \"maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:03.460025 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.460003    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-metrics-certs\") pod \"maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:03.466252 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.466227    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqwzk\" (UniqueName: \"kubernetes.io/projected/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-kube-api-access-fqwzk\") pod \"maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:03.963429 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.963392    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-maas-api-tls\") pod \"maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:03.965839 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:03.965809    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-maas-api-tls\") pod \"maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:04.224239 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:04.223912    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:04.505483 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:04.505381    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"]
Sep 09 06:12:04.508175 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:12:04.508143    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c92ba0e_3026_4b51_bd6c_a183c0ab6eb3.slice/crio-da6bb08895296e336a013825ab93814cee27841436d0550a23cd5f4357a04536 WatchSource:0}: Error finding container da6bb08895296e336a013825ab93814cee27841436d0550a23cd5f4357a04536: Status 404 returned error can't find the container with id da6bb08895296e336a013825ab93814cee27841436d0550a23cd5f4357a04536
Sep 09 06:12:04.509760 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:04.509738    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 06:12:05.359137 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:05.359097    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz" event={"ID":"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3","Type":"ContainerStarted","Data":"297cd8173ec36de5d76e2c25873394546606b686d813e4fd0cd9bf01bff406ee"}
Sep 09 06:12:05.359320 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:05.359257    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:05.359320 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:05.359295    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz" event={"ID":"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3","Type":"ContainerStarted","Data":"da6bb08895296e336a013825ab93814cee27841436d0550a23cd5f4357a04536"}
Sep 09 06:12:05.383960 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:05.383903    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz" podStartSLOduration=1.895307967 podStartE2EDuration="2.383887323s" totalImagesPullingTime="488.579356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:12:05.383080754 +0000 UTC m=+1352.599410639" watchObservedRunningTime="2026-09-09 06:12:05.383887323 +0000 UTC m=+1352.600217207"
Sep 09 06:12:11.371890 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:11.371860    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:12:32.277381 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.277348    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"]
Sep 09 06:12:32.282302 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.282278    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:12:32.285180 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.285156    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d907ff-serving-cert\""
Sep 09 06:12:32.293165 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.293137    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"]
Sep 09 06:12:32.461999 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.461961    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92gmd\" (UniqueName: \"kubernetes.io/projected/c660fd60-9455-434a-9b3c-d72806c9e459-kube-api-access-92gmd\") pod \"maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:12:32.462203 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.462025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c660fd60-9455-434a-9b3c-d72806c9e459-openshift-service-ca\") pod \"maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:12:32.462203 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.462132    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c660fd60-9455-434a-9b3c-d72806c9e459-metrics-certs\") pod \"maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:12:32.462203 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.462165    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c660fd60-9455-434a-9b3c-d72806c9e459-maas-api-tls\") pod \"maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:12:32.563090 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.563008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c660fd60-9455-434a-9b3c-d72806c9e459-metrics-certs\") pod \"maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:12:32.563090 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.563046    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c660fd60-9455-434a-9b3c-d72806c9e459-maas-api-tls\") pod \"maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:12:32.563603 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.563120    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92gmd\" (UniqueName: \"kubernetes.io/projected/c660fd60-9455-434a-9b3c-d72806c9e459-kube-api-access-92gmd\") pod \"maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:12:32.563603 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.563177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c660fd60-9455-434a-9b3c-d72806c9e459-openshift-service-ca\") pod \"maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:12:32.563879 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.563859    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c660fd60-9455-434a-9b3c-d72806c9e459-openshift-service-ca\") pod \"maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:12:32.565520 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.565480    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c660fd60-9455-434a-9b3c-d72806c9e459-maas-api-tls\") pod \"maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:12:32.565619 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.565602    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c660fd60-9455-434a-9b3c-d72806c9e459-metrics-certs\") pod \"maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:12:32.571519 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.571466    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92gmd\" (UniqueName: \"kubernetes.io/projected/c660fd60-9455-434a-9b3c-d72806c9e459-kube-api-access-92gmd\") pod \"maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:12:32.608395 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.608371    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:12:32.889426 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:32.889390    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"]
Sep 09 06:12:32.891735 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:12:32.891705    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc660fd60_9455_434a_9b3c_d72806c9e459.slice/crio-0c81b680c27d29ba7f2393665a4865e5d2a41834959ddafcb852dacf40d4001e WatchSource:0}: Error finding container 0c81b680c27d29ba7f2393665a4865e5d2a41834959ddafcb852dacf40d4001e: Status 404 returned error can't find the container with id 0c81b680c27d29ba7f2393665a4865e5d2a41834959ddafcb852dacf40d4001e
Sep 09 06:12:33.548584 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:33.548476    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q" event={"ID":"c660fd60-9455-434a-9b3c-d72806c9e459","Type":"ContainerStarted","Data":"a700e1959130e63aac1ff933d2c2f5349694e40b62c9654a86c8b77df13b496f"}
Sep 09 06:12:33.548584 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:33.548555    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q" event={"ID":"c660fd60-9455-434a-9b3c-d72806c9e459","Type":"ContainerStarted","Data":"0c81b680c27d29ba7f2393665a4865e5d2a41834959ddafcb852dacf40d4001e"}
Sep 09 06:12:33.548991 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:33.548698    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:12:33.566181 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:33.566133    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q" podStartSLOduration=1.17577008 podStartE2EDuration="1.566115501s" totalImagesPullingTime="390.345421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:12:33.562645993 +0000 UTC m=+1380.778975878" watchObservedRunningTime="2026-09-09 06:12:33.566115501 +0000 UTC m=+1380.782445384"
Sep 09 06:12:39.561257 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:12:39.561225    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:13:14.862906 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:14.862873    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"]
Sep 09 06:13:14.863427 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:14.863165    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz" podUID="7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3" containerName="maas-api" containerID="cri-o://297cd8173ec36de5d76e2c25873394546606b686d813e4fd0cd9bf01bff406ee" gracePeriod=30
Sep 09 06:13:15.120051 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.119971    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:13:15.201896 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.201866    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-maas-api-tls\") pod \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") "
Sep 09 06:13:15.202087 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.201937    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-openshift-service-ca\") pod \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") "
Sep 09 06:13:15.202087 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.201984    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-metrics-certs\") pod \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") "
Sep 09 06:13:15.202087 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.202072    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqwzk\" (UniqueName: \"kubernetes.io/projected/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-kube-api-access-fqwzk\") pod \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\" (UID: \"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3\") "
Sep 09 06:13:15.202336 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.202308    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3" (UID: "7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:13:15.202525 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.202447    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:15.204447 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.204406    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3" (UID: "7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:13:15.204683 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.204657    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3" (UID: "7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:13:15.204758 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.204679    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-kube-api-access-fqwzk" (OuterVolumeSpecName: "kube-api-access-fqwzk") pod "7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3" (UID: "7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3"). InnerVolumeSpecName "kube-api-access-fqwzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:13:15.303892 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.303859    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:15.303892 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.303885    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqwzk\" (UniqueName: \"kubernetes.io/projected/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-kube-api-access-fqwzk\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:15.303892 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.303894    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:15.822306 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.822269    2564 generic.go:356] "Generic (PLEG): container finished" podID="7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3" containerID="297cd8173ec36de5d76e2c25873394546606b686d813e4fd0cd9bf01bff406ee" exitCode=0
Sep 09 06:13:15.822515 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.822321    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"
Sep 09 06:13:15.822515 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.822321    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz" event={"ID":"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3","Type":"ContainerDied","Data":"297cd8173ec36de5d76e2c25873394546606b686d813e4fd0cd9bf01bff406ee"}
Sep 09 06:13:15.822515 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.822379    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz" event={"ID":"7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3","Type":"ContainerDied","Data":"da6bb08895296e336a013825ab93814cee27841436d0550a23cd5f4357a04536"}
Sep 09 06:13:15.822515 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.822407    2564 scope.go:117] "RemoveContainer" containerID="297cd8173ec36de5d76e2c25873394546606b686d813e4fd0cd9bf01bff406ee"
Sep 09 06:13:15.832127 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.832107    2564 scope.go:117] "RemoveContainer" containerID="297cd8173ec36de5d76e2c25873394546606b686d813e4fd0cd9bf01bff406ee"
Sep 09 06:13:15.832411 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:13:15.832394    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"297cd8173ec36de5d76e2c25873394546606b686d813e4fd0cd9bf01bff406ee\": container with ID starting with 297cd8173ec36de5d76e2c25873394546606b686d813e4fd0cd9bf01bff406ee not found: ID does not exist" containerID="297cd8173ec36de5d76e2c25873394546606b686d813e4fd0cd9bf01bff406ee"
Sep 09 06:13:15.832459 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.832419    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"297cd8173ec36de5d76e2c25873394546606b686d813e4fd0cd9bf01bff406ee"} err="failed to get container status \"297cd8173ec36de5d76e2c25873394546606b686d813e4fd0cd9bf01bff406ee\": rpc error: code = NotFound desc = could not find container \"297cd8173ec36de5d76e2c25873394546606b686d813e4fd0cd9bf01bff406ee\": container with ID starting with 297cd8173ec36de5d76e2c25873394546606b686d813e4fd0cd9bf01bff406ee not found: ID does not exist"
Sep 09 06:13:15.844883 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.844845    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"]
Sep 09 06:13:15.852925 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:15.852895    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e1ed02-6bd997595b-cxwzz"]
Sep 09 06:13:17.312508 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:17.312466    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3" path="/var/lib/kubelet/pods/7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3/volumes"
Sep 09 06:13:24.696937 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:24.696904    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"]
Sep 09 06:13:35.649654 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:35.649614    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"]
Sep 09 06:13:35.650175 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:35.649953    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq" podUID="bfe57a46-2294-467f-93a2-d1386a262c69" containerName="payload-processing" containerID="cri-o://64c43228fa96b953683802fcf435666fae09dca50b8e7ccd54794cffc9150e84" gracePeriod=30
Sep 09 06:13:35.657156 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:35.657131    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"]
Sep 09 06:13:35.657433 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:35.657375    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf" podUID="94d23818-c30b-4b0a-8469-3575669b2bf9" containerName="payload-pre-processing" containerID="cri-o://c62615a329e75a9b969459a8d95557be3f1fd9f8f95811d5dd4576721ec86bf7" gracePeriod=30
Sep 09 06:13:35.943456 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:35.943422    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"
Sep 09 06:13:35.955874 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:35.955848    2564 generic.go:356] "Generic (PLEG): container finished" podID="94d23818-c30b-4b0a-8469-3575669b2bf9" containerID="c62615a329e75a9b969459a8d95557be3f1fd9f8f95811d5dd4576721ec86bf7" exitCode=0
Sep 09 06:13:35.956004 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:35.955923    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf" event={"ID":"94d23818-c30b-4b0a-8469-3575669b2bf9","Type":"ContainerDied","Data":"c62615a329e75a9b969459a8d95557be3f1fd9f8f95811d5dd4576721ec86bf7"}
Sep 09 06:13:35.957587 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:35.957558    2564 generic.go:356] "Generic (PLEG): container finished" podID="bfe57a46-2294-467f-93a2-d1386a262c69" containerID="64c43228fa96b953683802fcf435666fae09dca50b8e7ccd54794cffc9150e84" exitCode=0
Sep 09 06:13:35.957701 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:35.957608    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq" event={"ID":"bfe57a46-2294-467f-93a2-d1386a262c69","Type":"ContainerDied","Data":"64c43228fa96b953683802fcf435666fae09dca50b8e7ccd54794cffc9150e84"}
Sep 09 06:13:35.957701 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:35.957629    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"
Sep 09 06:13:35.957701 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:35.957675    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq" event={"ID":"bfe57a46-2294-467f-93a2-d1386a262c69","Type":"ContainerDied","Data":"528363aedbaa6b0b925514c8280c6df331a834e6375d9657bb364578ce06d68f"}
Sep 09 06:13:35.957878 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:35.957707    2564 scope.go:117] "RemoveContainer" containerID="64c43228fa96b953683802fcf435666fae09dca50b8e7ccd54794cffc9150e84"
Sep 09 06:13:35.966272 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:35.966251    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"
Sep 09 06:13:35.968379 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:35.968346    2564 scope.go:117] "RemoveContainer" containerID="64c43228fa96b953683802fcf435666fae09dca50b8e7ccd54794cffc9150e84"
Sep 09 06:13:35.968687 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:13:35.968667    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c43228fa96b953683802fcf435666fae09dca50b8e7ccd54794cffc9150e84\": container with ID starting with 64c43228fa96b953683802fcf435666fae09dca50b8e7ccd54794cffc9150e84 not found: ID does not exist" containerID="64c43228fa96b953683802fcf435666fae09dca50b8e7ccd54794cffc9150e84"
Sep 09 06:13:35.968807 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:35.968694    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c43228fa96b953683802fcf435666fae09dca50b8e7ccd54794cffc9150e84"} err="failed to get container status \"64c43228fa96b953683802fcf435666fae09dca50b8e7ccd54794cffc9150e84\": rpc error: code = NotFound desc = could not find container \"64c43228fa96b953683802fcf435666fae09dca50b8e7ccd54794cffc9150e84\": container with ID starting with 64c43228fa96b953683802fcf435666fae09dca50b8e7ccd54794cffc9150e84 not found: ID does not exist"
Sep 09 06:13:36.133880 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.133835    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d23818-c30b-4b0a-8469-3575669b2bf9-plugins-config-volume\") pod \"94d23818-c30b-4b0a-8469-3575669b2bf9\" (UID: \"94d23818-c30b-4b0a-8469-3575669b2bf9\") "
Sep 09 06:13:36.133880 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.133883    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmvs6\" (UniqueName: \"kubernetes.io/projected/bfe57a46-2294-467f-93a2-d1386a262c69-kube-api-access-mmvs6\") pod \"bfe57a46-2294-467f-93a2-d1386a262c69\" (UID: \"bfe57a46-2294-467f-93a2-d1386a262c69\") "
Sep 09 06:13:36.134084 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.133930    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfvlg\" (UniqueName: \"kubernetes.io/projected/94d23818-c30b-4b0a-8469-3575669b2bf9-kube-api-access-vfvlg\") pod \"94d23818-c30b-4b0a-8469-3575669b2bf9\" (UID: \"94d23818-c30b-4b0a-8469-3575669b2bf9\") "
Sep 09 06:13:36.134084 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.134050    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe57a46-2294-467f-93a2-d1386a262c69-plugins-config-volume\") pod \"bfe57a46-2294-467f-93a2-d1386a262c69\" (UID: \"bfe57a46-2294-467f-93a2-d1386a262c69\") "
Sep 09 06:13:36.134214 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.134189    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d23818-c30b-4b0a-8469-3575669b2bf9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94d23818-c30b-4b0a-8469-3575669b2bf9" (UID: "94d23818-c30b-4b0a-8469-3575669b2bf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:13:36.134465 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.134435    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe57a46-2294-467f-93a2-d1386a262c69-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bfe57a46-2294-467f-93a2-d1386a262c69" (UID: "bfe57a46-2294-467f-93a2-d1386a262c69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:13:36.134465 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.134460    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d23818-c30b-4b0a-8469-3575669b2bf9-plugins-config-volume\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:36.136056 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.136034    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe57a46-2294-467f-93a2-d1386a262c69-kube-api-access-mmvs6" (OuterVolumeSpecName: "kube-api-access-mmvs6") pod "bfe57a46-2294-467f-93a2-d1386a262c69" (UID: "bfe57a46-2294-467f-93a2-d1386a262c69"). InnerVolumeSpecName "kube-api-access-mmvs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:13:36.136159 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.136061    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d23818-c30b-4b0a-8469-3575669b2bf9-kube-api-access-vfvlg" (OuterVolumeSpecName: "kube-api-access-vfvlg") pod "94d23818-c30b-4b0a-8469-3575669b2bf9" (UID: "94d23818-c30b-4b0a-8469-3575669b2bf9"). InnerVolumeSpecName "kube-api-access-vfvlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:13:36.235053 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.234975    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe57a46-2294-467f-93a2-d1386a262c69-plugins-config-volume\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:36.235053 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.235000    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmvs6\" (UniqueName: \"kubernetes.io/projected/bfe57a46-2294-467f-93a2-d1386a262c69-kube-api-access-mmvs6\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:36.235053 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.235011    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfvlg\" (UniqueName: \"kubernetes.io/projected/94d23818-c30b-4b0a-8469-3575669b2bf9-kube-api-access-vfvlg\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:36.280187 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.280157    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"]
Sep 09 06:13:36.285581 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.285556    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-43eb2d-687c76d8cf-z5qlq"]
Sep 09 06:13:36.966240 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.966157    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"
Sep 09 06:13:36.966240 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.966185    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf" event={"ID":"94d23818-c30b-4b0a-8469-3575669b2bf9","Type":"ContainerDied","Data":"088d61a5706970cfa328fb9eeff8bb8b9d9db1c33edaa6dcdf46d75786fd0ae3"}
Sep 09 06:13:36.966240 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.966236    2564 scope.go:117] "RemoveContainer" containerID="c62615a329e75a9b969459a8d95557be3f1fd9f8f95811d5dd4576721ec86bf7"
Sep 09 06:13:36.988086 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.988058    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"]
Sep 09 06:13:36.993587 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:36.993566    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-43eb2d-86669b4c74-2psmf"]
Sep 09 06:13:37.304034 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.303954    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94d23818-c30b-4b0a-8469-3575669b2bf9" path="/var/lib/kubelet/pods/94d23818-c30b-4b0a-8469-3575669b2bf9/volumes"
Sep 09 06:13:37.304329 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.304316    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfe57a46-2294-467f-93a2-d1386a262c69" path="/var/lib/kubelet/pods/bfe57a46-2294-467f-93a2-d1386a262c69/volumes"
Sep 09 06:13:37.397972 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.397938    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"]
Sep 09 06:13:37.398227 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.398202    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q" podUID="c660fd60-9455-434a-9b3c-d72806c9e459" containerName="maas-api" containerID="cri-o://a700e1959130e63aac1ff933d2c2f5349694e40b62c9654a86c8b77df13b496f" gracePeriod=30
Sep 09 06:13:37.649333 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.649302    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:13:37.750076 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.750042    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92gmd\" (UniqueName: \"kubernetes.io/projected/c660fd60-9455-434a-9b3c-d72806c9e459-kube-api-access-92gmd\") pod \"c660fd60-9455-434a-9b3c-d72806c9e459\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") "
Sep 09 06:13:37.750229 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.750106    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c660fd60-9455-434a-9b3c-d72806c9e459-maas-api-tls\") pod \"c660fd60-9455-434a-9b3c-d72806c9e459\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") "
Sep 09 06:13:37.750229 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.750137    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c660fd60-9455-434a-9b3c-d72806c9e459-openshift-service-ca\") pod \"c660fd60-9455-434a-9b3c-d72806c9e459\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") "
Sep 09 06:13:37.750314 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.750249    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c660fd60-9455-434a-9b3c-d72806c9e459-metrics-certs\") pod \"c660fd60-9455-434a-9b3c-d72806c9e459\" (UID: \"c660fd60-9455-434a-9b3c-d72806c9e459\") "
Sep 09 06:13:37.750724 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.750696    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c660fd60-9455-434a-9b3c-d72806c9e459-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c660fd60-9455-434a-9b3c-d72806c9e459" (UID: "c660fd60-9455-434a-9b3c-d72806c9e459"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:13:37.752620 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.752597    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c660fd60-9455-434a-9b3c-d72806c9e459-kube-api-access-92gmd" (OuterVolumeSpecName: "kube-api-access-92gmd") pod "c660fd60-9455-434a-9b3c-d72806c9e459" (UID: "c660fd60-9455-434a-9b3c-d72806c9e459"). InnerVolumeSpecName "kube-api-access-92gmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:13:37.752918 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.752893    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c660fd60-9455-434a-9b3c-d72806c9e459-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c660fd60-9455-434a-9b3c-d72806c9e459" (UID: "c660fd60-9455-434a-9b3c-d72806c9e459"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:13:37.752990 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.752974    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c660fd60-9455-434a-9b3c-d72806c9e459-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c660fd60-9455-434a-9b3c-d72806c9e459" (UID: "c660fd60-9455-434a-9b3c-d72806c9e459"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:13:37.851567 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.851532    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92gmd\" (UniqueName: \"kubernetes.io/projected/c660fd60-9455-434a-9b3c-d72806c9e459-kube-api-access-92gmd\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:37.851567 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.851566    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c660fd60-9455-434a-9b3c-d72806c9e459-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:37.851753 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.851579    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c660fd60-9455-434a-9b3c-d72806c9e459-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:37.851753 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.851589    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c660fd60-9455-434a-9b3c-d72806c9e459-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:37.974743 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.974712    2564 generic.go:356] "Generic (PLEG): container finished" podID="c660fd60-9455-434a-9b3c-d72806c9e459" containerID="a700e1959130e63aac1ff933d2c2f5349694e40b62c9654a86c8b77df13b496f" exitCode=0
Sep 09 06:13:37.975183 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.974757    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"
Sep 09 06:13:37.975183 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.974799    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q" event={"ID":"c660fd60-9455-434a-9b3c-d72806c9e459","Type":"ContainerDied","Data":"a700e1959130e63aac1ff933d2c2f5349694e40b62c9654a86c8b77df13b496f"}
Sep 09 06:13:37.975183 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.974853    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q" event={"ID":"c660fd60-9455-434a-9b3c-d72806c9e459","Type":"ContainerDied","Data":"0c81b680c27d29ba7f2393665a4865e5d2a41834959ddafcb852dacf40d4001e"}
Sep 09 06:13:37.975183 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.974892    2564 scope.go:117] "RemoveContainer" containerID="a700e1959130e63aac1ff933d2c2f5349694e40b62c9654a86c8b77df13b496f"
Sep 09 06:13:37.985312 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.985295    2564 scope.go:117] "RemoveContainer" containerID="a700e1959130e63aac1ff933d2c2f5349694e40b62c9654a86c8b77df13b496f"
Sep 09 06:13:37.985614 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:13:37.985596    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a700e1959130e63aac1ff933d2c2f5349694e40b62c9654a86c8b77df13b496f\": container with ID starting with a700e1959130e63aac1ff933d2c2f5349694e40b62c9654a86c8b77df13b496f not found: ID does not exist" containerID="a700e1959130e63aac1ff933d2c2f5349694e40b62c9654a86c8b77df13b496f"
Sep 09 06:13:37.985676 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.985623    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a700e1959130e63aac1ff933d2c2f5349694e40b62c9654a86c8b77df13b496f"} err="failed to get container status \"a700e1959130e63aac1ff933d2c2f5349694e40b62c9654a86c8b77df13b496f\": rpc error: code = NotFound desc = could not find container \"a700e1959130e63aac1ff933d2c2f5349694e40b62c9654a86c8b77df13b496f\": container with ID starting with a700e1959130e63aac1ff933d2c2f5349694e40b62c9654a86c8b77df13b496f not found: ID does not exist"
Sep 09 06:13:37.996642 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:37.996607    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"]
Sep 09 06:13:38.001199 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:38.001171    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d907ff-bdf67668d-8jl8q"]
Sep 09 06:13:39.303095 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:39.303060    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c660fd60-9455-434a-9b3c-d72806c9e459" path="/var/lib/kubelet/pods/c660fd60-9455-434a-9b3c-d72806c9e459/volumes"
Sep 09 06:13:39.717691 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:39.717652    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp" podUID="132aa2fd-ce5b-4146-9d07-ebacfa2663e5" containerName="main" containerID="cri-o://c54db14705d04b1d6c28e6d327b89b37a3d8dba32aaa8da6e86e8e33ed91fa65" gracePeriod=45
Sep 09 06:13:39.980767 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:39.980703    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:13:39.999507 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:39.999414    2564 generic.go:356] "Generic (PLEG): container finished" podID="132aa2fd-ce5b-4146-9d07-ebacfa2663e5" containerID="c54db14705d04b1d6c28e6d327b89b37a3d8dba32aaa8da6e86e8e33ed91fa65" exitCode=0
Sep 09 06:13:39.999700 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:39.999525    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp" event={"ID":"132aa2fd-ce5b-4146-9d07-ebacfa2663e5","Type":"ContainerDied","Data":"c54db14705d04b1d6c28e6d327b89b37a3d8dba32aaa8da6e86e8e33ed91fa65"}
Sep 09 06:13:39.999700 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:39.999572    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp" event={"ID":"132aa2fd-ce5b-4146-9d07-ebacfa2663e5","Type":"ContainerDied","Data":"ab9e261648d41f5dc134cc47666e774e8adc60944757042f74ca28f8083ee34a"}
Sep 09 06:13:39.999700 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:39.999594    2564 scope.go:117] "RemoveContainer" containerID="c54db14705d04b1d6c28e6d327b89b37a3d8dba32aaa8da6e86e8e33ed91fa65"
Sep 09 06:13:39.999700 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:39.999631    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"
Sep 09 06:13:40.011419 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.011393    2564 scope.go:117] "RemoveContainer" containerID="c54db14705d04b1d6c28e6d327b89b37a3d8dba32aaa8da6e86e8e33ed91fa65"
Sep 09 06:13:40.011802 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:13:40.011777    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c54db14705d04b1d6c28e6d327b89b37a3d8dba32aaa8da6e86e8e33ed91fa65\": container with ID starting with c54db14705d04b1d6c28e6d327b89b37a3d8dba32aaa8da6e86e8e33ed91fa65 not found: ID does not exist" containerID="c54db14705d04b1d6c28e6d327b89b37a3d8dba32aaa8da6e86e8e33ed91fa65"
Sep 09 06:13:40.011890 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.011810    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54db14705d04b1d6c28e6d327b89b37a3d8dba32aaa8da6e86e8e33ed91fa65"} err="failed to get container status \"c54db14705d04b1d6c28e6d327b89b37a3d8dba32aaa8da6e86e8e33ed91fa65\": rpc error: code = NotFound desc = could not find container \"c54db14705d04b1d6c28e6d327b89b37a3d8dba32aaa8da6e86e8e33ed91fa65\": container with ID starting with c54db14705d04b1d6c28e6d327b89b37a3d8dba32aaa8da6e86e8e33ed91fa65 not found: ID does not exist"
Sep 09 06:13:40.072231 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.072199    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-dshm\") pod \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") "
Sep 09 06:13:40.072401 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.072245    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-tls-certs\") pod \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") "
Sep 09 06:13:40.072401 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.072290    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5r2z\" (UniqueName: \"kubernetes.io/projected/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-kube-api-access-h5r2z\") pod \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") "
Sep 09 06:13:40.072401 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.072306    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-home\") pod \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") "
Sep 09 06:13:40.072401 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.072338    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-model-cache\") pod \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") "
Sep 09 06:13:40.072647 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.072406    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-tmp-dir\") pod \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\" (UID: \"132aa2fd-ce5b-4146-9d07-ebacfa2663e5\") "
Sep 09 06:13:40.072742 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.072674    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-home" (OuterVolumeSpecName: "home") pod "132aa2fd-ce5b-4146-9d07-ebacfa2663e5" (UID: "132aa2fd-ce5b-4146-9d07-ebacfa2663e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.072742 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.072728    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "132aa2fd-ce5b-4146-9d07-ebacfa2663e5" (UID: "132aa2fd-ce5b-4146-9d07-ebacfa2663e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.072866 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.072806    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "132aa2fd-ce5b-4146-9d07-ebacfa2663e5" (UID: "132aa2fd-ce5b-4146-9d07-ebacfa2663e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.074842 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.074818    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "132aa2fd-ce5b-4146-9d07-ebacfa2663e5" (UID: "132aa2fd-ce5b-4146-9d07-ebacfa2663e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:13:40.074947 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.074885    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-kube-api-access-h5r2z" (OuterVolumeSpecName: "kube-api-access-h5r2z") pod "132aa2fd-ce5b-4146-9d07-ebacfa2663e5" (UID: "132aa2fd-ce5b-4146-9d07-ebacfa2663e5"). InnerVolumeSpecName "kube-api-access-h5r2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:13:40.075589 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.075567    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-dshm" (OuterVolumeSpecName: "dshm") pod "132aa2fd-ce5b-4146-9d07-ebacfa2663e5" (UID: "132aa2fd-ce5b-4146-9d07-ebacfa2663e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.173760 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.173731    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-dshm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.173760 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.173758    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-tls-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.173962 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.173772    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5r2z\" (UniqueName: \"kubernetes.io/projected/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-kube-api-access-h5r2z\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.173962 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.173784    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-home\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.173962 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.173795    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-model-cache\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.173962 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.173807    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/132aa2fd-ce5b-4146-9d07-ebacfa2663e5-tmp-dir\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.322188 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.322154    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"]
Sep 09 06:13:40.327572 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:40.327537    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/unconfigured-w4-43eb2d-kserve-788fb4f5b-c7mhp"]
Sep 09 06:13:41.303609 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:41.303576    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="132aa2fd-ce5b-4146-9d07-ebacfa2663e5" path="/var/lib/kubelet/pods/132aa2fd-ce5b-4146-9d07-ebacfa2663e5/volumes"
Sep 09 06:13:55.075430 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:55.075387    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"]
Sep 09 06:13:55.077853 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:55.075664    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc" podUID="f1c34dc6-f827-420b-aa07-0ceb7535dad5" containerName="maas-api" containerID="cri-o://96c9ecae783a4938d2f7823ed0a797fd446d3e28edb7040a3fddc3ca063ba610" gracePeriod=30
Sep 09 06:13:55.340747 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:55.340675    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:13:55.431609 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:55.431576    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c34dc6-f827-420b-aa07-0ceb7535dad5-maas-api-tls\") pod \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") "
Sep 09 06:13:55.431777 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:55.431756    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c34dc6-f827-420b-aa07-0ceb7535dad5-metrics-certs\") pod \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") "
Sep 09 06:13:55.431829 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:55.431800    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg745\" (UniqueName: \"kubernetes.io/projected/f1c34dc6-f827-420b-aa07-0ceb7535dad5-kube-api-access-vg745\") pod \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") "
Sep 09 06:13:55.431829 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:55.431825    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c34dc6-f827-420b-aa07-0ceb7535dad5-openshift-service-ca\") pod \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\" (UID: \"f1c34dc6-f827-420b-aa07-0ceb7535dad5\") "
Sep 09 06:13:55.432240 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:55.432203    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c34dc6-f827-420b-aa07-0ceb7535dad5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f1c34dc6-f827-420b-aa07-0ceb7535dad5" (UID: "f1c34dc6-f827-420b-aa07-0ceb7535dad5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:13:55.433781 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:55.433753    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c34dc6-f827-420b-aa07-0ceb7535dad5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f1c34dc6-f827-420b-aa07-0ceb7535dad5" (UID: "f1c34dc6-f827-420b-aa07-0ceb7535dad5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:13:55.433892 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:55.433878    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c34dc6-f827-420b-aa07-0ceb7535dad5-kube-api-access-vg745" (OuterVolumeSpecName: "kube-api-access-vg745") pod "f1c34dc6-f827-420b-aa07-0ceb7535dad5" (UID: "f1c34dc6-f827-420b-aa07-0ceb7535dad5"). InnerVolumeSpecName "kube-api-access-vg745". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:13:55.433960 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:55.433941    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c34dc6-f827-420b-aa07-0ceb7535dad5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f1c34dc6-f827-420b-aa07-0ceb7535dad5" (UID: "f1c34dc6-f827-420b-aa07-0ceb7535dad5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:13:55.533377 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:55.533342    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c34dc6-f827-420b-aa07-0ceb7535dad5-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:55.533377 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:55.533370    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg745\" (UniqueName: \"kubernetes.io/projected/f1c34dc6-f827-420b-aa07-0ceb7535dad5-kube-api-access-vg745\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:55.533377 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:55.533381    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c34dc6-f827-420b-aa07-0ceb7535dad5-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:55.533645 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:55.533390    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c34dc6-f827-420b-aa07-0ceb7535dad5-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:13:56.105635 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:56.105595    2564 generic.go:356] "Generic (PLEG): container finished" podID="f1c34dc6-f827-420b-aa07-0ceb7535dad5" containerID="96c9ecae783a4938d2f7823ed0a797fd446d3e28edb7040a3fddc3ca063ba610" exitCode=0
Sep 09 06:13:56.106038 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:56.105647    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"
Sep 09 06:13:56.106038 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:56.105680    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc" event={"ID":"f1c34dc6-f827-420b-aa07-0ceb7535dad5","Type":"ContainerDied","Data":"96c9ecae783a4938d2f7823ed0a797fd446d3e28edb7040a3fddc3ca063ba610"}
Sep 09 06:13:56.106038 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:56.105735    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc" event={"ID":"f1c34dc6-f827-420b-aa07-0ceb7535dad5","Type":"ContainerDied","Data":"346e0ed9ac12de89efa100233df2e4b8dcb1b74cf91d7e2ac5416b405a3a894c"}
Sep 09 06:13:56.106038 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:56.105760    2564 scope.go:117] "RemoveContainer" containerID="96c9ecae783a4938d2f7823ed0a797fd446d3e28edb7040a3fddc3ca063ba610"
Sep 09 06:13:56.115202 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:56.115185    2564 scope.go:117] "RemoveContainer" containerID="96c9ecae783a4938d2f7823ed0a797fd446d3e28edb7040a3fddc3ca063ba610"
Sep 09 06:13:56.115438 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:13:56.115419    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c9ecae783a4938d2f7823ed0a797fd446d3e28edb7040a3fddc3ca063ba610\": container with ID starting with 96c9ecae783a4938d2f7823ed0a797fd446d3e28edb7040a3fddc3ca063ba610 not found: ID does not exist" containerID="96c9ecae783a4938d2f7823ed0a797fd446d3e28edb7040a3fddc3ca063ba610"
Sep 09 06:13:56.115479 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:56.115446    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c9ecae783a4938d2f7823ed0a797fd446d3e28edb7040a3fddc3ca063ba610"} err="failed to get container status \"96c9ecae783a4938d2f7823ed0a797fd446d3e28edb7040a3fddc3ca063ba610\": rpc error: code = NotFound desc = could not find container \"96c9ecae783a4938d2f7823ed0a797fd446d3e28edb7040a3fddc3ca063ba610\": container with ID starting with 96c9ecae783a4938d2f7823ed0a797fd446d3e28edb7040a3fddc3ca063ba610 not found: ID does not exist"
Sep 09 06:13:56.128425 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:56.128405    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"]
Sep 09 06:13:56.135224 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:56.135194    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f3d736-848cd58bc9-dctvc"]
Sep 09 06:13:57.302655 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:13:57.302618    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1c34dc6-f827-420b-aa07-0ceb7535dad5" path="/var/lib/kubelet/pods/f1c34dc6-f827-420b-aa07-0ceb7535dad5/volumes"
Sep 09 06:14:14.352564 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:14.352532    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"]
Sep 09 06:14:14.352990 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:14.352752    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9" podUID="321b3854-6df4-469f-8b5a-c75de70c685e" containerName="maas-api" containerID="cri-o://5a2dbe4ba3f16b98737a9598dcbbd526daf7ecbbd0aa8c02b5aa05983093ee47" gracePeriod=30
Sep 09 06:14:14.620970 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:14.620939    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:14:14.711838 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:14.711810    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321b3854-6df4-469f-8b5a-c75de70c685e-maas-api-tls\") pod \"321b3854-6df4-469f-8b5a-c75de70c685e\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") "
Sep 09 06:14:14.712028 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:14.711870    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f4qd\" (UniqueName: \"kubernetes.io/projected/321b3854-6df4-469f-8b5a-c75de70c685e-kube-api-access-5f4qd\") pod \"321b3854-6df4-469f-8b5a-c75de70c685e\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") "
Sep 09 06:14:14.712028 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:14.711939    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321b3854-6df4-469f-8b5a-c75de70c685e-metrics-certs\") pod \"321b3854-6df4-469f-8b5a-c75de70c685e\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") "
Sep 09 06:14:14.712028 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:14.711971    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321b3854-6df4-469f-8b5a-c75de70c685e-openshift-service-ca\") pod \"321b3854-6df4-469f-8b5a-c75de70c685e\" (UID: \"321b3854-6df4-469f-8b5a-c75de70c685e\") "
Sep 09 06:14:14.712350 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:14.712324    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321b3854-6df4-469f-8b5a-c75de70c685e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "321b3854-6df4-469f-8b5a-c75de70c685e" (UID: "321b3854-6df4-469f-8b5a-c75de70c685e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:14:14.714044 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:14.714016    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/321b3854-6df4-469f-8b5a-c75de70c685e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "321b3854-6df4-469f-8b5a-c75de70c685e" (UID: "321b3854-6df4-469f-8b5a-c75de70c685e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:14:14.714154 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:14.714077    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321b3854-6df4-469f-8b5a-c75de70c685e-kube-api-access-5f4qd" (OuterVolumeSpecName: "kube-api-access-5f4qd") pod "321b3854-6df4-469f-8b5a-c75de70c685e" (UID: "321b3854-6df4-469f-8b5a-c75de70c685e"). InnerVolumeSpecName "kube-api-access-5f4qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:14:14.714154 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:14.714115    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/321b3854-6df4-469f-8b5a-c75de70c685e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "321b3854-6df4-469f-8b5a-c75de70c685e" (UID: "321b3854-6df4-469f-8b5a-c75de70c685e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:14:14.813147 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:14.813095    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321b3854-6df4-469f-8b5a-c75de70c685e-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:14:14.813147 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:14.813145    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f4qd\" (UniqueName: \"kubernetes.io/projected/321b3854-6df4-469f-8b5a-c75de70c685e-kube-api-access-5f4qd\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:14:14.813147 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:14.813160    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321b3854-6df4-469f-8b5a-c75de70c685e-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:14:14.813381 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:14.813174    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321b3854-6df4-469f-8b5a-c75de70c685e-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:14:15.234789 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:15.234749    2564 generic.go:356] "Generic (PLEG): container finished" podID="321b3854-6df4-469f-8b5a-c75de70c685e" containerID="5a2dbe4ba3f16b98737a9598dcbbd526daf7ecbbd0aa8c02b5aa05983093ee47" exitCode=0
Sep 09 06:14:15.234964 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:15.234811    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"
Sep 09 06:14:15.234964 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:15.234834    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9" event={"ID":"321b3854-6df4-469f-8b5a-c75de70c685e","Type":"ContainerDied","Data":"5a2dbe4ba3f16b98737a9598dcbbd526daf7ecbbd0aa8c02b5aa05983093ee47"}
Sep 09 06:14:15.234964 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:15.234912    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9" event={"ID":"321b3854-6df4-469f-8b5a-c75de70c685e","Type":"ContainerDied","Data":"b1901a7e4abaab91d9f2a967c7351dcbfd4f1ae8cc373febf3ecf63473cb973d"}
Sep 09 06:14:15.234964 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:15.234933    2564 scope.go:117] "RemoveContainer" containerID="5a2dbe4ba3f16b98737a9598dcbbd526daf7ecbbd0aa8c02b5aa05983093ee47"
Sep 09 06:14:15.245591 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:15.245572    2564 scope.go:117] "RemoveContainer" containerID="5a2dbe4ba3f16b98737a9598dcbbd526daf7ecbbd0aa8c02b5aa05983093ee47"
Sep 09 06:14:15.245911 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:14:15.245890    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a2dbe4ba3f16b98737a9598dcbbd526daf7ecbbd0aa8c02b5aa05983093ee47\": container with ID starting with 5a2dbe4ba3f16b98737a9598dcbbd526daf7ecbbd0aa8c02b5aa05983093ee47 not found: ID does not exist" containerID="5a2dbe4ba3f16b98737a9598dcbbd526daf7ecbbd0aa8c02b5aa05983093ee47"
Sep 09 06:14:15.245979 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:15.245918    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a2dbe4ba3f16b98737a9598dcbbd526daf7ecbbd0aa8c02b5aa05983093ee47"} err="failed to get container status \"5a2dbe4ba3f16b98737a9598dcbbd526daf7ecbbd0aa8c02b5aa05983093ee47\": rpc error: code = NotFound desc = could not find container \"5a2dbe4ba3f16b98737a9598dcbbd526daf7ecbbd0aa8c02b5aa05983093ee47\": container with ID starting with 5a2dbe4ba3f16b98737a9598dcbbd526daf7ecbbd0aa8c02b5aa05983093ee47 not found: ID does not exist"
Sep 09 06:14:15.260043 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:15.260014    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"]
Sep 09 06:14:15.263047 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:15.263021    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-750c68-7f55c8dc88-wzdb9"]
Sep 09 06:14:15.303107 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:15.303069    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="321b3854-6df4-469f-8b5a-c75de70c685e" path="/var/lib/kubelet/pods/321b3854-6df4-469f-8b5a-c75de70c685e/volumes"
Sep 09 06:14:21.637362 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:21.637327    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"]
Sep 09 06:14:21.637890 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:21.637603    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj" podUID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerName="istio-proxy" containerID="cri-o://3d7d2d9aa72a8dab48aaddf5539209ef6b37a983086c78db9bd9f3a9e9f377ff" gracePeriod=30
Sep 09 06:14:21.665840 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:21.665801    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 06:14:21.665987 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:21.665863    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj" podUID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 06:14:26.896686 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.896658    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:14:26.939302 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.939225    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-workload-socket\") pod \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") "
Sep 09 06:14:26.939302 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.939286    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-envoy\") pod \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") "
Sep 09 06:14:26.939551 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.939356    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-data\") pod \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") "
Sep 09 06:14:26.939551 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.939403    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-workload-certs\") pod \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") "
Sep 09 06:14:26.939551 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.939470    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-podinfo\") pod \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") "
Sep 09 06:14:26.939551 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.939518    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-token\") pod \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") "
Sep 09 06:14:26.939551 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.939547    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-credential-socket\") pod \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") "
Sep 09 06:14:26.939841 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.939572    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istiod-ca-cert\") pod \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") "
Sep 09 06:14:26.939841 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.939560    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" (UID: "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:14:26.939841 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.939605    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wmzw\" (UniqueName: \"kubernetes.io/projected/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-kube-api-access-9wmzw\") pod \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\" (UID: \"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f\") "
Sep 09 06:14:26.940008 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.939875    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" (UID: "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:14:26.940062 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.940011    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-data\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:14:26.940062 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.940032    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-workload-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:14:26.940245 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.940224    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" (UID: "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:14:26.941788 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.941758    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" (UID: "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:14:26.941909 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.941886    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" (UID: "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:14:26.942148 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.942117    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" (UID: "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:14:26.942148 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.942142    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-kube-api-access-9wmzw" (OuterVolumeSpecName: "kube-api-access-9wmzw") pod "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" (UID: "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f"). InnerVolumeSpecName "kube-api-access-9wmzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:14:26.942532 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.942510    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" (UID: "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:14:26.942665 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:26.942625    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" (UID: "cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 06:14:27.041372 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.041341    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-podinfo\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:14:27.041372 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.041368    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-token\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:14:27.041372 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.041378    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-credential-socket\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:14:27.041626 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.041386    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istiod-ca-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:14:27.041626 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.041395    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wmzw\" (UniqueName: \"kubernetes.io/projected/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-kube-api-access-9wmzw\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:14:27.041626 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.041403    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-istio-envoy\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:14:27.041626 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.041411    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f-workload-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:14:27.310385 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.310304    2564 generic.go:356] "Generic (PLEG): container finished" podID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerID="3d7d2d9aa72a8dab48aaddf5539209ef6b37a983086c78db9bd9f3a9e9f377ff" exitCode=0
Sep 09 06:14:27.310713 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.310409    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"
Sep 09 06:14:27.310713 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.310421    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj" event={"ID":"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f","Type":"ContainerDied","Data":"3d7d2d9aa72a8dab48aaddf5539209ef6b37a983086c78db9bd9f3a9e9f377ff"}
Sep 09 06:14:27.310713 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.310476    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj" event={"ID":"cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f","Type":"ContainerDied","Data":"20384d97911aab4d2fe915e3cba8b5d46af706278467a7aff524d5a558fc388d"}
Sep 09 06:14:27.310713 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.310524    2564 scope.go:117] "RemoveContainer" containerID="3d7d2d9aa72a8dab48aaddf5539209ef6b37a983086c78db9bd9f3a9e9f377ff"
Sep 09 06:14:27.320123 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.320110    2564 scope.go:117] "RemoveContainer" containerID="11cef41c04716d70dd62e77976c5adf97318fd77e1db4ea87bc72b1b3c308e06"
Sep 09 06:14:27.329418 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.329387    2564 scope.go:117] "RemoveContainer" containerID="3d7d2d9aa72a8dab48aaddf5539209ef6b37a983086c78db9bd9f3a9e9f377ff"
Sep 09 06:14:27.329792 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:14:27.329772    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d7d2d9aa72a8dab48aaddf5539209ef6b37a983086c78db9bd9f3a9e9f377ff\": container with ID starting with 3d7d2d9aa72a8dab48aaddf5539209ef6b37a983086c78db9bd9f3a9e9f377ff not found: ID does not exist" containerID="3d7d2d9aa72a8dab48aaddf5539209ef6b37a983086c78db9bd9f3a9e9f377ff"
Sep 09 06:14:27.329876 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.329802    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d7d2d9aa72a8dab48aaddf5539209ef6b37a983086c78db9bd9f3a9e9f377ff"} err="failed to get container status \"3d7d2d9aa72a8dab48aaddf5539209ef6b37a983086c78db9bd9f3a9e9f377ff\": rpc error: code = NotFound desc = could not find container \"3d7d2d9aa72a8dab48aaddf5539209ef6b37a983086c78db9bd9f3a9e9f377ff\": container with ID starting with 3d7d2d9aa72a8dab48aaddf5539209ef6b37a983086c78db9bd9f3a9e9f377ff not found: ID does not exist"
Sep 09 06:14:27.329876 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.329819    2564 scope.go:117] "RemoveContainer" containerID="11cef41c04716d70dd62e77976c5adf97318fd77e1db4ea87bc72b1b3c308e06"
Sep 09 06:14:27.330097 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:14:27.330079    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11cef41c04716d70dd62e77976c5adf97318fd77e1db4ea87bc72b1b3c308e06\": container with ID starting with 11cef41c04716d70dd62e77976c5adf97318fd77e1db4ea87bc72b1b3c308e06 not found: ID does not exist" containerID="11cef41c04716d70dd62e77976c5adf97318fd77e1db4ea87bc72b1b3c308e06"
Sep 09 06:14:27.330181 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.330106    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cef41c04716d70dd62e77976c5adf97318fd77e1db4ea87bc72b1b3c308e06"} err="failed to get container status \"11cef41c04716d70dd62e77976c5adf97318fd77e1db4ea87bc72b1b3c308e06\": rpc error: code = NotFound desc = could not find container \"11cef41c04716d70dd62e77976c5adf97318fd77e1db4ea87bc72b1b3c308e06\": container with ID starting with 11cef41c04716d70dd62e77976c5adf97318fd77e1db4ea87bc72b1b3c308e06 not found: ID does not exist"
Sep 09 06:14:27.330981 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.330843    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"]
Sep 09 06:14:27.334350 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:27.334330    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-750c68-openshift-default-bc9b8bbf8-n9vtj"]
Sep 09 06:14:29.303940 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:29.303905    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" path="/var/lib/kubelet/pods/cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f/volumes"
Sep 09 06:14:33.405037 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:33.405010    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 06:14:33.417410 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:33.417388    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 06:14:55.019302 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.019266    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"]
Sep 09 06:14:55.019969 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.019944    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132aa2fd-ce5b-4146-9d07-ebacfa2663e5" containerName="main"
Sep 09 06:14:55.020138 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020118    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="132aa2fd-ce5b-4146-9d07-ebacfa2663e5" containerName="main"
Sep 09 06:14:55.020252 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020146    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94d23818-c30b-4b0a-8469-3575669b2bf9" containerName="payload-pre-processing"
Sep 09 06:14:55.020252 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020156    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="94d23818-c30b-4b0a-8469-3575669b2bf9" containerName="payload-pre-processing"
Sep 09 06:14:55.020252 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020172    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="321b3854-6df4-469f-8b5a-c75de70c685e" containerName="maas-api"
Sep 09 06:14:55.020252 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020180    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="321b3854-6df4-469f-8b5a-c75de70c685e" containerName="maas-api"
Sep 09 06:14:55.020252 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020206    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c34dc6-f827-420b-aa07-0ceb7535dad5" containerName="maas-api"
Sep 09 06:14:55.020252 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020215    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c34dc6-f827-420b-aa07-0ceb7535dad5" containerName="maas-api"
Sep 09 06:14:55.020252 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020233    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c660fd60-9455-434a-9b3c-d72806c9e459" containerName="maas-api"
Sep 09 06:14:55.020252 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020240    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c660fd60-9455-434a-9b3c-d72806c9e459" containerName="maas-api"
Sep 09 06:14:55.020672 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020257    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerName="istio-proxy"
Sep 09 06:14:55.020672 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020265    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerName="istio-proxy"
Sep 09 06:14:55.020672 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020274    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3" containerName="maas-api"
Sep 09 06:14:55.020672 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020281    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3" containerName="maas-api"
Sep 09 06:14:55.020672 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020299    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe57a46-2294-467f-93a2-d1386a262c69" containerName="payload-processing"
Sep 09 06:14:55.020672 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020307    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe57a46-2294-467f-93a2-d1386a262c69" containerName="payload-processing"
Sep 09 06:14:55.020672 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020316    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerName="istio-proxy"
Sep 09 06:14:55.020672 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020324    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerName="istio-proxy"
Sep 09 06:14:55.020672 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020442    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c34dc6-f827-420b-aa07-0ceb7535dad5" containerName="maas-api"
Sep 09 06:14:55.020672 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020457    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c660fd60-9455-434a-9b3c-d72806c9e459" containerName="maas-api"
Sep 09 06:14:55.020672 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020468    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="132aa2fd-ce5b-4146-9d07-ebacfa2663e5" containerName="main"
Sep 09 06:14:55.020672 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020478    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerName="istio-proxy"
Sep 09 06:14:55.021207 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020487    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe57a46-2294-467f-93a2-d1386a262c69" containerName="payload-processing"
Sep 09 06:14:55.021207 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020764    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="94d23818-c30b-4b0a-8469-3575669b2bf9" containerName="payload-pre-processing"
Sep 09 06:14:55.021207 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020783    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="321b3854-6df4-469f-8b5a-c75de70c685e" containerName="maas-api"
Sep 09 06:14:55.021207 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.020792    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c92ba0e-3026-4b51-bd6c-a183c0ab6eb3" containerName="maas-api"
Sep 09 06:14:55.024272 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.024251    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.027850 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.027827    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7cf6be\"/\"sub-test-model-7cf6be-kserve-self-signed-certs\""
Sep 09 06:14:55.028504 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.027828    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7cf6be\"/\"default-dockercfg-7wv4f\""
Sep 09 06:14:55.028504 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.027919    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7cf6be\"/\"kube-root-ca.crt\""
Sep 09 06:14:55.028504 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.027921    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7cf6be\"/\"openshift-service-ca.crt\""
Sep 09 06:14:55.032302 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.032281    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"]
Sep 09 06:14:55.112253 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.112210    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-model-cache\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.112431 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.112280    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-dshm\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.112431 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.112346    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-home\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.112431 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.112412    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b5a650-3c90-4084-967a-4702f3a8b4ca-tls-certs\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.112597 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.112437    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbsgn\" (UniqueName: \"kubernetes.io/projected/05b5a650-3c90-4084-967a-4702f3a8b4ca-kube-api-access-sbsgn\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.112597 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.112519    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-tmp-dir\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.213802 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.213764    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b5a650-3c90-4084-967a-4702f3a8b4ca-tls-certs\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.213802 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.213803    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbsgn\" (UniqueName: \"kubernetes.io/projected/05b5a650-3c90-4084-967a-4702f3a8b4ca-kube-api-access-sbsgn\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.214026 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.213850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-tmp-dir\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.214026 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.213879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-model-cache\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.214026 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.213928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-dshm\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.214026 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.213962    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-home\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.214359 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.214336    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-tmp-dir\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.214434 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.214352    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-model-cache\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.214434 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.214394    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-home\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.216146 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.216124    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-dshm\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.216290 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.216273    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b5a650-3c90-4084-967a-4702f3a8b4ca-tls-certs\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.221614 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.221594    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbsgn\" (UniqueName: \"kubernetes.io/projected/05b5a650-3c90-4084-967a-4702f3a8b4ca-kube-api-access-sbsgn\") pod \"sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.335654 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.335627    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:14:55.609039 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:55.609012    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"]
Sep 09 06:14:55.611051 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:14:55.611012    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05b5a650_3c90_4084_967a_4702f3a8b4ca.slice/crio-9e3a0e79a4556e2208daaafb72e73ee1defba208d23db02760997564c696e88e WatchSource:0}: Error finding container 9e3a0e79a4556e2208daaafb72e73ee1defba208d23db02760997564c696e88e: Status 404 returned error can't find the container with id 9e3a0e79a4556e2208daaafb72e73ee1defba208d23db02760997564c696e88e
Sep 09 06:14:56.483432 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:56.483392    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds" event={"ID":"05b5a650-3c90-4084-967a-4702f3a8b4ca","Type":"ContainerStarted","Data":"dde38117767c4f8cd9686d323e4515dbe7c7768fb1c4ee3210af3dfedf4bb0c9"}
Sep 09 06:14:56.483840 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:56.483444    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds" event={"ID":"05b5a650-3c90-4084-967a-4702f3a8b4ca","Type":"ContainerStarted","Data":"9e3a0e79a4556e2208daaafb72e73ee1defba208d23db02760997564c696e88e"}
Sep 09 06:14:56.502340 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:14:56.502288    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds" podStartSLOduration=1.502273629 podStartE2EDuration="1.502273629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:14:56.50047753 +0000 UTC m=+1523.716807414" watchObservedRunningTime="2026-09-09 06:14:56.502273629 +0000 UTC m=+1523.718603513"
Sep 09 06:15:05.336258 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:05.336224    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:15:05.336653 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:05.336271    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:15:05.348844 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:05.348815    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:15:05.545866 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:05.545834    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:15:31.343060 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:31.343024    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"]
Sep 09 06:15:46.364309 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.364246    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds" podUID="05b5a650-3c90-4084-967a-4702f3a8b4ca" containerName="main" containerID="cri-o://dde38117767c4f8cd9686d323e4515dbe7c7768fb1c4ee3210af3dfedf4bb0c9" gracePeriod=45
Sep 09 06:15:46.612891 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.612864    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:15:46.715430 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.715353    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b5a650-3c90-4084-967a-4702f3a8b4ca-tls-certs\") pod \"05b5a650-3c90-4084-967a-4702f3a8b4ca\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") "
Sep 09 06:15:46.715430 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.715394    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-tmp-dir\") pod \"05b5a650-3c90-4084-967a-4702f3a8b4ca\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") "
Sep 09 06:15:46.715430 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.715424    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-model-cache\") pod \"05b5a650-3c90-4084-967a-4702f3a8b4ca\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") "
Sep 09 06:15:46.715693 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.715457    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-home\") pod \"05b5a650-3c90-4084-967a-4702f3a8b4ca\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") "
Sep 09 06:15:46.715693 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.715535    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-dshm\") pod \"05b5a650-3c90-4084-967a-4702f3a8b4ca\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") "
Sep 09 06:15:46.715693 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.715567    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbsgn\" (UniqueName: \"kubernetes.io/projected/05b5a650-3c90-4084-967a-4702f3a8b4ca-kube-api-access-sbsgn\") pod \"05b5a650-3c90-4084-967a-4702f3a8b4ca\" (UID: \"05b5a650-3c90-4084-967a-4702f3a8b4ca\") "
Sep 09 06:15:46.715844 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.715710    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05b5a650-3c90-4084-967a-4702f3a8b4ca" (UID: "05b5a650-3c90-4084-967a-4702f3a8b4ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:15:46.715844 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.715725    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "05b5a650-3c90-4084-967a-4702f3a8b4ca" (UID: "05b5a650-3c90-4084-967a-4702f3a8b4ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:15:46.715940 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.715854    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-home" (OuterVolumeSpecName: "home") pod "05b5a650-3c90-4084-967a-4702f3a8b4ca" (UID: "05b5a650-3c90-4084-967a-4702f3a8b4ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:15:46.715940 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.715922    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-tmp-dir\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:15:46.715940 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.715936    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-model-cache\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:15:46.717781 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.717746    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b5a650-3c90-4084-967a-4702f3a8b4ca-kube-api-access-sbsgn" (OuterVolumeSpecName: "kube-api-access-sbsgn") pod "05b5a650-3c90-4084-967a-4702f3a8b4ca" (UID: "05b5a650-3c90-4084-967a-4702f3a8b4ca"). InnerVolumeSpecName "kube-api-access-sbsgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:15:46.717781 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.717764    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-dshm" (OuterVolumeSpecName: "dshm") pod "05b5a650-3c90-4084-967a-4702f3a8b4ca" (UID: "05b5a650-3c90-4084-967a-4702f3a8b4ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:15:46.717924 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.717827    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05b5a650-3c90-4084-967a-4702f3a8b4ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05b5a650-3c90-4084-967a-4702f3a8b4ca" (UID: "05b5a650-3c90-4084-967a-4702f3a8b4ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:15:46.793296 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.793245    2564 generic.go:356] "Generic (PLEG): container finished" podID="05b5a650-3c90-4084-967a-4702f3a8b4ca" containerID="dde38117767c4f8cd9686d323e4515dbe7c7768fb1c4ee3210af3dfedf4bb0c9" exitCode=0
Sep 09 06:15:46.793296 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.793282    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"
Sep 09 06:15:46.793525 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.793304    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds" event={"ID":"05b5a650-3c90-4084-967a-4702f3a8b4ca","Type":"ContainerDied","Data":"dde38117767c4f8cd9686d323e4515dbe7c7768fb1c4ee3210af3dfedf4bb0c9"}
Sep 09 06:15:46.793525 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.793349    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds" event={"ID":"05b5a650-3c90-4084-967a-4702f3a8b4ca","Type":"ContainerDied","Data":"9e3a0e79a4556e2208daaafb72e73ee1defba208d23db02760997564c696e88e"}
Sep 09 06:15:46.793525 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.793372    2564 scope.go:117] "RemoveContainer" containerID="dde38117767c4f8cd9686d323e4515dbe7c7768fb1c4ee3210af3dfedf4bb0c9"
Sep 09 06:15:46.803351 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.803334    2564 scope.go:117] "RemoveContainer" containerID="dde38117767c4f8cd9686d323e4515dbe7c7768fb1c4ee3210af3dfedf4bb0c9"
Sep 09 06:15:46.803694 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:15:46.803670    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde38117767c4f8cd9686d323e4515dbe7c7768fb1c4ee3210af3dfedf4bb0c9\": container with ID starting with dde38117767c4f8cd9686d323e4515dbe7c7768fb1c4ee3210af3dfedf4bb0c9 not found: ID does not exist" containerID="dde38117767c4f8cd9686d323e4515dbe7c7768fb1c4ee3210af3dfedf4bb0c9"
Sep 09 06:15:46.803772 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.803707    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde38117767c4f8cd9686d323e4515dbe7c7768fb1c4ee3210af3dfedf4bb0c9"} err="failed to get container status \"dde38117767c4f8cd9686d323e4515dbe7c7768fb1c4ee3210af3dfedf4bb0c9\": rpc error: code = NotFound desc = could not find container \"dde38117767c4f8cd9686d323e4515dbe7c7768fb1c4ee3210af3dfedf4bb0c9\": container with ID starting with dde38117767c4f8cd9686d323e4515dbe7c7768fb1c4ee3210af3dfedf4bb0c9 not found: ID does not exist"
Sep 09 06:15:46.815398 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.815376    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"]
Sep 09 06:15:46.817094 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.817069    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbsgn\" (UniqueName: \"kubernetes.io/projected/05b5a650-3c90-4084-967a-4702f3a8b4ca-kube-api-access-sbsgn\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:15:46.817175 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.817098    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b5a650-3c90-4084-967a-4702f3a8b4ca-tls-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:15:46.817175 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.817112    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-home\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:15:46.817175 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.817123    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b5a650-3c90-4084-967a-4702f3a8b4ca-dshm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:15:46.819604 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:46.819584    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/sub-test-model-7cf6be-kserve-84b5ff6b9-ftcds"]
Sep 09 06:15:47.302561 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:15:47.302525    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05b5a650-3c90-4084-967a-4702f3a8b4ca" path="/var/lib/kubelet/pods/05b5a650-3c90-4084-967a-4702f3a8b4ca/volumes"
Sep 09 06:16:09.121208 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.121171    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"]
Sep 09 06:16:09.121811 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.121648    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05b5a650-3c90-4084-967a-4702f3a8b4ca" containerName="main"
Sep 09 06:16:09.121811 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.121662    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="05b5a650-3c90-4084-967a-4702f3a8b4ca" containerName="main"
Sep 09 06:16:09.121811 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.121734    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="05b5a650-3c90-4084-967a-4702f3a8b4ca" containerName="main"
Sep 09 06:16:09.121811 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.121745    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd3eeac3-b584-4fb6-8c0d-0cb42ad3152f" containerName="istio-proxy"
Sep 09 06:16:09.124799 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.124781    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.127747 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.127723    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7cf6be\"/\"kube-root-ca.crt\""
Sep 09 06:16:09.128598 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.128581    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7cf6be\"/\"openshift-service-ca.crt\""
Sep 09 06:16:09.128659 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.128610    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7cf6be\"/\"default-dockercfg-7wv4f\""
Sep 09 06:16:09.128695 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.128678    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7cf6be\"/\"rate-test-model-7cf6be-kserve-self-signed-certs\""
Sep 09 06:16:09.132095 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.131007    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkqfk\" (UniqueName: \"kubernetes.io/projected/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-kube-api-access-qkqfk\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.132095 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.131083    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-tmp-dir\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.132095 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.131114    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-tls-certs\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.132095 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.131236    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-home\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.132095 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.131281    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-dshm\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.132095 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.131306    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-model-cache\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.135430 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.135286    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"]
Sep 09 06:16:09.232044 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.232005    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-tmp-dir\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.232044 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.232057    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-tls-certs\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.232312 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.232154    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-home\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.232312 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.232201    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-dshm\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.232312 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.232228    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-model-cache\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.232312 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.232286    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkqfk\" (UniqueName: \"kubernetes.io/projected/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-kube-api-access-qkqfk\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.232554 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.232535    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-home\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.232703 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.232679    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-tmp-dir\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.232815 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.232699    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-model-cache\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.234863 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.234838    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-dshm\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.235287 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.235264    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-tls-certs\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.241283 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.241262    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkqfk\" (UniqueName: \"kubernetes.io/projected/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-kube-api-access-qkqfk\") pod \"rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.439909 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.439829    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:09.712078 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.712051    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"]
Sep 09 06:16:09.713561 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:16:09.713530    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78dd53a7_5b4d_462d_99bd_16f094dcf5dd.slice/crio-7ac175e50d23b106854390c48f94115be1571f9995ddbb1d21953c6e76426e43 WatchSource:0}: Error finding container 7ac175e50d23b106854390c48f94115be1571f9995ddbb1d21953c6e76426e43: Status 404 returned error can't find the container with id 7ac175e50d23b106854390c48f94115be1571f9995ddbb1d21953c6e76426e43
Sep 09 06:16:09.934781 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.934741    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t" event={"ID":"78dd53a7-5b4d-462d-99bd-16f094dcf5dd","Type":"ContainerStarted","Data":"43402d29c881c47cc732f0b3f5ce9ef09fc04b26e5731335867009a277270f0b"}
Sep 09 06:16:09.934781 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.934793    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t" event={"ID":"78dd53a7-5b4d-462d-99bd-16f094dcf5dd","Type":"ContainerStarted","Data":"7ac175e50d23b106854390c48f94115be1571f9995ddbb1d21953c6e76426e43"}
Sep 09 06:16:09.951817 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:09.951772    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t" podStartSLOduration=0.951759304 podStartE2EDuration="951.759304ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:16:09.951513429 +0000 UTC m=+1597.167843309" watchObservedRunningTime="2026-09-09 06:16:09.951759304 +0000 UTC m=+1597.168089187"
Sep 09 06:16:19.440734 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:19.440693    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:19.440734 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:19.440744    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:19.452891 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:19.452864    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:20.004655 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:20.004626    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:16:46.116018 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:46.115983    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"]
Sep 09 06:16:51.291175 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.291143    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"]
Sep 09 06:16:51.295371 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.295349    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:16:51.297772 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.297753    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8a6cf7-serving-cert\""
Sep 09 06:16:51.306675 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.306648    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"]
Sep 09 06:16:51.423856 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.423820    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mss76\" (UniqueName: \"kubernetes.io/projected/4a9e76fd-3082-45c8-957f-09677c83be84-kube-api-access-mss76\") pod \"maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:16:51.424038 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.423888    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a9e76fd-3082-45c8-957f-09677c83be84-metrics-certs\") pod \"maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:16:51.424038 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.423938    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9e76fd-3082-45c8-957f-09677c83be84-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:16:51.424125 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.424054    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a9e76fd-3082-45c8-957f-09677c83be84-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:16:51.524913 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.524882    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a9e76fd-3082-45c8-957f-09677c83be84-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:16:51.525093 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.524935    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mss76\" (UniqueName: \"kubernetes.io/projected/4a9e76fd-3082-45c8-957f-09677c83be84-kube-api-access-mss76\") pod \"maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:16:51.525093 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.524971    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a9e76fd-3082-45c8-957f-09677c83be84-metrics-certs\") pod \"maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:16:51.525093 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.524987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9e76fd-3082-45c8-957f-09677c83be84-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:16:51.525651 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.525627    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9e76fd-3082-45c8-957f-09677c83be84-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:16:51.527338 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.527313    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a9e76fd-3082-45c8-957f-09677c83be84-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:16:51.527419 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.527362    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a9e76fd-3082-45c8-957f-09677c83be84-metrics-certs\") pod \"maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:16:51.533283 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.533264    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mss76\" (UniqueName: \"kubernetes.io/projected/4a9e76fd-3082-45c8-957f-09677c83be84-kube-api-access-mss76\") pod \"maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:16:51.609926 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.609895    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:16:51.889008 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:51.888979    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"]
Sep 09 06:16:51.890592 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:16:51.890569    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9e76fd_3082_45c8_957f_09677c83be84.slice/crio-f4473a0b6807b7c02ba104d1b1fb761a234ffb462a0d3d31bea38b1a1ae0f90f WatchSource:0}: Error finding container f4473a0b6807b7c02ba104d1b1fb761a234ffb462a0d3d31bea38b1a1ae0f90f: Status 404 returned error can't find the container with id f4473a0b6807b7c02ba104d1b1fb761a234ffb462a0d3d31bea38b1a1ae0f90f
Sep 09 06:16:52.189916 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:52.189831    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t" event={"ID":"4a9e76fd-3082-45c8-957f-09677c83be84","Type":"ContainerStarted","Data":"f4473a0b6807b7c02ba104d1b1fb761a234ffb462a0d3d31bea38b1a1ae0f90f"}
Sep 09 06:16:53.197314 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:53.197280    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t" event={"ID":"4a9e76fd-3082-45c8-957f-09677c83be84","Type":"ContainerStarted","Data":"df271b1f207e9931fdde7799d8c98eb54a09a920fb4a5646a3cd6b9eef2ce3e5"}
Sep 09 06:16:53.197717 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:53.197390    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:16:53.214233 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:53.214171    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t" podStartSLOduration=1.764618294 podStartE2EDuration="2.214156847s" totalImagesPullingTime="449.538553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:16:53.213027444 +0000 UTC m=+1640.429357341" watchObservedRunningTime="2026-09-09 06:16:53.214156847 +0000 UTC m=+1640.430486731"
Sep 09 06:16:59.209019 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:16:59.208990    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:17:01.135852 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.135803    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t" podUID="78dd53a7-5b4d-462d-99bd-16f094dcf5dd" containerName="main" containerID="cri-o://43402d29c881c47cc732f0b3f5ce9ef09fc04b26e5731335867009a277270f0b" gracePeriod=45
Sep 09 06:17:01.391882 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.391820    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:17:01.420274 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.420243    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-tmp-dir\") pod \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") "
Sep 09 06:17:01.420446 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.420327    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkqfk\" (UniqueName: \"kubernetes.io/projected/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-kube-api-access-qkqfk\") pod \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") "
Sep 09 06:17:01.420446 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.420350    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-model-cache\") pod \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") "
Sep 09 06:17:01.420446 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.420403    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-dshm\") pod \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") "
Sep 09 06:17:01.420645 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.420451    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-home\") pod \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") "
Sep 09 06:17:01.420645 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.420510    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-tls-certs\") pod \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\" (UID: \"78dd53a7-5b4d-462d-99bd-16f094dcf5dd\") "
Sep 09 06:17:01.420813 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.420783    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-home" (OuterVolumeSpecName: "home") pod "78dd53a7-5b4d-462d-99bd-16f094dcf5dd" (UID: "78dd53a7-5b4d-462d-99bd-16f094dcf5dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:17:01.420915 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.420886    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78dd53a7-5b4d-462d-99bd-16f094dcf5dd" (UID: "78dd53a7-5b4d-462d-99bd-16f094dcf5dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:17:01.421054 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.420945    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "78dd53a7-5b4d-462d-99bd-16f094dcf5dd" (UID: "78dd53a7-5b4d-462d-99bd-16f094dcf5dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:17:01.421126 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.420947    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-home\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:17:01.422735 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.422710    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-dshm" (OuterVolumeSpecName: "dshm") pod "78dd53a7-5b4d-462d-99bd-16f094dcf5dd" (UID: "78dd53a7-5b4d-462d-99bd-16f094dcf5dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:17:01.422832 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.422749    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78dd53a7-5b4d-462d-99bd-16f094dcf5dd" (UID: "78dd53a7-5b4d-462d-99bd-16f094dcf5dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:17:01.423127 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.423106    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-kube-api-access-qkqfk" (OuterVolumeSpecName: "kube-api-access-qkqfk") pod "78dd53a7-5b4d-462d-99bd-16f094dcf5dd" (UID: "78dd53a7-5b4d-462d-99bd-16f094dcf5dd"). InnerVolumeSpecName "kube-api-access-qkqfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:17:01.522363 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.522321    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkqfk\" (UniqueName: \"kubernetes.io/projected/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-kube-api-access-qkqfk\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:17:01.522363 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.522356    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-model-cache\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:17:01.522363 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.522365    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-dshm\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:17:01.522643 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.522374    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-tls-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:17:01.522643 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:01.522382    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78dd53a7-5b4d-462d-99bd-16f094dcf5dd-tmp-dir\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:17:02.256097 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:02.256058    2564 generic.go:356] "Generic (PLEG): container finished" podID="78dd53a7-5b4d-462d-99bd-16f094dcf5dd" containerID="43402d29c881c47cc732f0b3f5ce9ef09fc04b26e5731335867009a277270f0b" exitCode=0
Sep 09 06:17:02.256520 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:02.256104    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"
Sep 09 06:17:02.256520 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:02.256142    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t" event={"ID":"78dd53a7-5b4d-462d-99bd-16f094dcf5dd","Type":"ContainerDied","Data":"43402d29c881c47cc732f0b3f5ce9ef09fc04b26e5731335867009a277270f0b"}
Sep 09 06:17:02.256520 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:02.256186    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t" event={"ID":"78dd53a7-5b4d-462d-99bd-16f094dcf5dd","Type":"ContainerDied","Data":"7ac175e50d23b106854390c48f94115be1571f9995ddbb1d21953c6e76426e43"}
Sep 09 06:17:02.256520 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:02.256206    2564 scope.go:117] "RemoveContainer" containerID="43402d29c881c47cc732f0b3f5ce9ef09fc04b26e5731335867009a277270f0b"
Sep 09 06:17:02.268542 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:02.268515    2564 scope.go:117] "RemoveContainer" containerID="43402d29c881c47cc732f0b3f5ce9ef09fc04b26e5731335867009a277270f0b"
Sep 09 06:17:02.268852 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:17:02.268833    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43402d29c881c47cc732f0b3f5ce9ef09fc04b26e5731335867009a277270f0b\": container with ID starting with 43402d29c881c47cc732f0b3f5ce9ef09fc04b26e5731335867009a277270f0b not found: ID does not exist" containerID="43402d29c881c47cc732f0b3f5ce9ef09fc04b26e5731335867009a277270f0b"
Sep 09 06:17:02.268904 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:02.268862    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43402d29c881c47cc732f0b3f5ce9ef09fc04b26e5731335867009a277270f0b"} err="failed to get container status \"43402d29c881c47cc732f0b3f5ce9ef09fc04b26e5731335867009a277270f0b\": rpc error: code = NotFound desc = could not find container \"43402d29c881c47cc732f0b3f5ce9ef09fc04b26e5731335867009a277270f0b\": container with ID starting with 43402d29c881c47cc732f0b3f5ce9ef09fc04b26e5731335867009a277270f0b not found: ID does not exist"
Sep 09 06:17:02.279075 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:02.279042    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"]
Sep 09 06:17:02.281637 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:02.281615    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/rate-test-model-7cf6be-kserve-5888b8cc79-hsg9t"]
Sep 09 06:17:03.303353 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:03.303321    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78dd53a7-5b4d-462d-99bd-16f094dcf5dd" path="/var/lib/kubelet/pods/78dd53a7-5b4d-462d-99bd-16f094dcf5dd/volumes"
Sep 09 06:17:19.752286 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:19.752251    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"]
Sep 09 06:17:19.752939 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:19.752917    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78dd53a7-5b4d-462d-99bd-16f094dcf5dd" containerName="main"
Sep 09 06:17:19.752939 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:19.752940    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="78dd53a7-5b4d-462d-99bd-16f094dcf5dd" containerName="main"
Sep 09 06:17:19.753108 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:19.753073    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="78dd53a7-5b4d-462d-99bd-16f094dcf5dd" containerName="main"
Sep 09 06:17:19.756412 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:19.756390    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:17:19.760173 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:19.760146    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-32ea4d-serving-cert\""
Sep 09 06:17:19.769080 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:19.769055    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"]
Sep 09 06:17:19.906556 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:19.906520    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nsjs\" (UniqueName: \"kubernetes.io/projected/2520c649-79c2-4776-96a9-0c9534517939-kube-api-access-5nsjs\") pod \"maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:17:19.906556 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:19.906557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2520c649-79c2-4776-96a9-0c9534517939-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:17:19.906821 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:19.906674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2520c649-79c2-4776-96a9-0c9534517939-maas-api-tls\") pod \"maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:17:19.906821 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:19.906738    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2520c649-79c2-4776-96a9-0c9534517939-metrics-certs\") pod \"maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:17:20.007554 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:20.007443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2520c649-79c2-4776-96a9-0c9534517939-metrics-certs\") pod \"maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:17:20.007554 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:20.007553    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nsjs\" (UniqueName: \"kubernetes.io/projected/2520c649-79c2-4776-96a9-0c9534517939-kube-api-access-5nsjs\") pod \"maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:17:20.007792 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:20.007577    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2520c649-79c2-4776-96a9-0c9534517939-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:17:20.007792 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:20.007623    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2520c649-79c2-4776-96a9-0c9534517939-maas-api-tls\") pod \"maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:17:20.008263 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:20.008239    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2520c649-79c2-4776-96a9-0c9534517939-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:17:20.009988 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:20.009959    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2520c649-79c2-4776-96a9-0c9534517939-maas-api-tls\") pod \"maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:17:20.010090 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:20.009990    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2520c649-79c2-4776-96a9-0c9534517939-metrics-certs\") pod \"maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:17:20.020310 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:20.020287    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nsjs\" (UniqueName: \"kubernetes.io/projected/2520c649-79c2-4776-96a9-0c9534517939-kube-api-access-5nsjs\") pod \"maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:17:20.072280 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:20.072257    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:17:20.352421 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:20.352397    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"]
Sep 09 06:17:20.353111 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:17:20.353085    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2520c649_79c2_4776_96a9_0c9534517939.slice/crio-4741fe4e1828173b264acdb87caece88a14e9309482dc17f7c963e8b36dee4a0 WatchSource:0}: Error finding container 4741fe4e1828173b264acdb87caece88a14e9309482dc17f7c963e8b36dee4a0: Status 404 returned error can't find the container with id 4741fe4e1828173b264acdb87caece88a14e9309482dc17f7c963e8b36dee4a0
Sep 09 06:17:20.354354 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:20.354337    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 06:17:20.365278 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:20.365254    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg" event={"ID":"2520c649-79c2-4776-96a9-0c9534517939","Type":"ContainerStarted","Data":"4741fe4e1828173b264acdb87caece88a14e9309482dc17f7c963e8b36dee4a0"}
Sep 09 06:17:21.372581 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:21.372543    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg" event={"ID":"2520c649-79c2-4776-96a9-0c9534517939","Type":"ContainerStarted","Data":"09e6a1e1eba11d7d812f1f7aeb1a1062aab20d97ef5d081eb6a9e26741918e11"}
Sep 09 06:17:21.372954 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:21.372718    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:17:21.391465 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:21.391403    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg" podStartSLOduration=2.075079891 podStartE2EDuration="2.391381517s" totalImagesPullingTime="316.301626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:17:21.387362825 +0000 UTC m=+1668.603692710" watchObservedRunningTime="2026-09-09 06:17:21.391381517 +0000 UTC m=+1668.607711405"
Sep 09 06:17:27.383764 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:17:27.383734    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:18:29.149300 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.149263    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"]
Sep 09 06:18:29.153077 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.153059    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:29.155602 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.155569    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-2e28ff-serving-cert\""
Sep 09 06:18:29.164250 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.164225    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"]
Sep 09 06:18:29.281884 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.281849    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1751498-77bf-4c05-bb4c-78a123946ec4-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:29.282094 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.281903    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1751498-77bf-4c05-bb4c-78a123946ec4-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:29.282094 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.281981    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1751498-77bf-4c05-bb4c-78a123946ec4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:29.282094 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.282054    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxcvb\" (UniqueName: \"kubernetes.io/projected/b1751498-77bf-4c05-bb4c-78a123946ec4-kube-api-access-dxcvb\") pod \"maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:29.383624 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.383586    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxcvb\" (UniqueName: \"kubernetes.io/projected/b1751498-77bf-4c05-bb4c-78a123946ec4-kube-api-access-dxcvb\") pod \"maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:29.383832 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.383676    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1751498-77bf-4c05-bb4c-78a123946ec4-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:29.383832 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.383705    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1751498-77bf-4c05-bb4c-78a123946ec4-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:29.383832 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.383782    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1751498-77bf-4c05-bb4c-78a123946ec4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:29.384006 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:18:29.383889    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-serving-cert: secret "maas-api-e2e-ipp-cleanup-2e28ff-serving-cert" not found
Sep 09 06:18:29.384006 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:18:29.383987    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1751498-77bf-4c05-bb4c-78a123946ec4-maas-api-tls podName:b1751498-77bf-4c05-bb4c-78a123946ec4 nodeName:}" failed. No retries permitted until 2026-09-09 06:18:29.883965124 +0000 UTC m=+1737.100294991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1751498-77bf-4c05-bb4c-78a123946ec4-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps" (UID: "b1751498-77bf-4c05-bb4c-78a123946ec4") : secret "maas-api-e2e-ipp-cleanup-2e28ff-serving-cert" not found
Sep 09 06:18:29.384746 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.384716    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1751498-77bf-4c05-bb4c-78a123946ec4-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:29.386553 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.386526    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1751498-77bf-4c05-bb4c-78a123946ec4-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:29.394657 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.394628    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxcvb\" (UniqueName: \"kubernetes.io/projected/b1751498-77bf-4c05-bb4c-78a123946ec4-kube-api-access-dxcvb\") pod \"maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:29.888234 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.888190    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1751498-77bf-4c05-bb4c-78a123946ec4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:29.890654 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:29.890627    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1751498-77bf-4c05-bb4c-78a123946ec4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:30.066652 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:30.066615    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:30.342931 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:30.342896    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"]
Sep 09 06:18:30.343790 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:18:30.343768    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1751498_77bf_4c05_bb4c_78a123946ec4.slice/crio-0f3b59117611946ed692e6b0ed393b9213da4eabad76287385b19b8884ff5b3c WatchSource:0}: Error finding container 0f3b59117611946ed692e6b0ed393b9213da4eabad76287385b19b8884ff5b3c: Status 404 returned error can't find the container with id 0f3b59117611946ed692e6b0ed393b9213da4eabad76287385b19b8884ff5b3c
Sep 09 06:18:30.805465 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:30.805433    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps" event={"ID":"b1751498-77bf-4c05-bb4c-78a123946ec4","Type":"ContainerStarted","Data":"0f3b59117611946ed692e6b0ed393b9213da4eabad76287385b19b8884ff5b3c"}
Sep 09 06:18:31.812844 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:31.812802    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps" event={"ID":"b1751498-77bf-4c05-bb4c-78a123946ec4","Type":"ContainerStarted","Data":"560bb1993d3d53e3dca501eee11368ce7913fa2e5cd259c2ad86bc16a0877c63"}
Sep 09 06:18:31.813219 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:31.812897    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:31.834898 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:31.834854    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps" podStartSLOduration=2.403080596 podStartE2EDuration="2.834839932s" totalImagesPullingTime="431.759336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:18:31.83261514 +0000 UTC m=+1739.048945024" watchObservedRunningTime="2026-09-09 06:18:31.834839932 +0000 UTC m=+1739.051169848"
Sep 09 06:18:37.824352 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:37.824321    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:59.655303 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.655271    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"]
Sep 09 06:18:59.657657 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.655462    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps" podUID="b1751498-77bf-4c05-bb4c-78a123946ec4" containerName="maas-api" containerID="cri-o://560bb1993d3d53e3dca501eee11368ce7913fa2e5cd259c2ad86bc16a0877c63" gracePeriod=30
Sep 09 06:18:59.920780 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.920719    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:59.977415 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.977375    2564 generic.go:356] "Generic (PLEG): container finished" podID="b1751498-77bf-4c05-bb4c-78a123946ec4" containerID="560bb1993d3d53e3dca501eee11368ce7913fa2e5cd259c2ad86bc16a0877c63" exitCode=0
Sep 09 06:18:59.977607 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.977424    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"
Sep 09 06:18:59.977607 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.977428    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps" event={"ID":"b1751498-77bf-4c05-bb4c-78a123946ec4","Type":"ContainerDied","Data":"560bb1993d3d53e3dca501eee11368ce7913fa2e5cd259c2ad86bc16a0877c63"}
Sep 09 06:18:59.977607 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.977479    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps" event={"ID":"b1751498-77bf-4c05-bb4c-78a123946ec4","Type":"ContainerDied","Data":"0f3b59117611946ed692e6b0ed393b9213da4eabad76287385b19b8884ff5b3c"}
Sep 09 06:18:59.977607 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.977519    2564 scope.go:117] "RemoveContainer" containerID="560bb1993d3d53e3dca501eee11368ce7913fa2e5cd259c2ad86bc16a0877c63"
Sep 09 06:18:59.982456 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.982431    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1751498-77bf-4c05-bb4c-78a123946ec4-openshift-service-ca\") pod \"b1751498-77bf-4c05-bb4c-78a123946ec4\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") "
Sep 09 06:18:59.982593 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.982533    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1751498-77bf-4c05-bb4c-78a123946ec4-metrics-certs\") pod \"b1751498-77bf-4c05-bb4c-78a123946ec4\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") "
Sep 09 06:18:59.982650 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.982616    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxcvb\" (UniqueName: \"kubernetes.io/projected/b1751498-77bf-4c05-bb4c-78a123946ec4-kube-api-access-dxcvb\") pod \"b1751498-77bf-4c05-bb4c-78a123946ec4\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") "
Sep 09 06:18:59.982707 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.982653    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1751498-77bf-4c05-bb4c-78a123946ec4-maas-api-tls\") pod \"b1751498-77bf-4c05-bb4c-78a123946ec4\" (UID: \"b1751498-77bf-4c05-bb4c-78a123946ec4\") "
Sep 09 06:18:59.982838 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.982812    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1751498-77bf-4c05-bb4c-78a123946ec4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b1751498-77bf-4c05-bb4c-78a123946ec4" (UID: "b1751498-77bf-4c05-bb4c-78a123946ec4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:18:59.983024 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.983007    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1751498-77bf-4c05-bb4c-78a123946ec4-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:18:59.984622 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.984601    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1751498-77bf-4c05-bb4c-78a123946ec4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b1751498-77bf-4c05-bb4c-78a123946ec4" (UID: "b1751498-77bf-4c05-bb4c-78a123946ec4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:18:59.984824 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.984806    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1751498-77bf-4c05-bb4c-78a123946ec4-kube-api-access-dxcvb" (OuterVolumeSpecName: "kube-api-access-dxcvb") pod "b1751498-77bf-4c05-bb4c-78a123946ec4" (UID: "b1751498-77bf-4c05-bb4c-78a123946ec4"). InnerVolumeSpecName "kube-api-access-dxcvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:18:59.984922 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.984865    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1751498-77bf-4c05-bb4c-78a123946ec4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b1751498-77bf-4c05-bb4c-78a123946ec4" (UID: "b1751498-77bf-4c05-bb4c-78a123946ec4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:18:59.987387 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.987366    2564 scope.go:117] "RemoveContainer" containerID="560bb1993d3d53e3dca501eee11368ce7913fa2e5cd259c2ad86bc16a0877c63"
Sep 09 06:18:59.987734 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:18:59.987714    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560bb1993d3d53e3dca501eee11368ce7913fa2e5cd259c2ad86bc16a0877c63\": container with ID starting with 560bb1993d3d53e3dca501eee11368ce7913fa2e5cd259c2ad86bc16a0877c63 not found: ID does not exist" containerID="560bb1993d3d53e3dca501eee11368ce7913fa2e5cd259c2ad86bc16a0877c63"
Sep 09 06:18:59.987807 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:18:59.987743    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560bb1993d3d53e3dca501eee11368ce7913fa2e5cd259c2ad86bc16a0877c63"} err="failed to get container status \"560bb1993d3d53e3dca501eee11368ce7913fa2e5cd259c2ad86bc16a0877c63\": rpc error: code = NotFound desc = could not find container \"560bb1993d3d53e3dca501eee11368ce7913fa2e5cd259c2ad86bc16a0877c63\": container with ID starting with 560bb1993d3d53e3dca501eee11368ce7913fa2e5cd259c2ad86bc16a0877c63 not found: ID does not exist"
Sep 09 06:19:00.084235 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:00.084196    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1751498-77bf-4c05-bb4c-78a123946ec4-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:19:00.084235 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:00.084233    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxcvb\" (UniqueName: \"kubernetes.io/projected/b1751498-77bf-4c05-bb4c-78a123946ec4-kube-api-access-dxcvb\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:19:00.084235 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:00.084242    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1751498-77bf-4c05-bb4c-78a123946ec4-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:19:00.305782 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:00.305749    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"]
Sep 09 06:19:00.310267 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:00.310239    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2e28ff-779656569d-r4rps"]
Sep 09 06:19:01.303040 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:01.303001    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1751498-77bf-4c05-bb4c-78a123946ec4" path="/var/lib/kubelet/pods/b1751498-77bf-4c05-bb4c-78a123946ec4/volumes"
Sep 09 06:19:25.524839 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:25.524795    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"]
Sep 09 06:19:25.525280 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:25.525036    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t" podUID="4a9e76fd-3082-45c8-957f-09677c83be84" containerName="maas-api" containerID="cri-o://df271b1f207e9931fdde7799d8c98eb54a09a920fb4a5646a3cd6b9eef2ce3e5" gracePeriod=30
Sep 09 06:19:25.784741 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:25.784678    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:19:25.938285 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:25.938252    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a9e76fd-3082-45c8-957f-09677c83be84-metrics-certs\") pod \"4a9e76fd-3082-45c8-957f-09677c83be84\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") "
Sep 09 06:19:25.938469 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:25.938304    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9e76fd-3082-45c8-957f-09677c83be84-openshift-service-ca\") pod \"4a9e76fd-3082-45c8-957f-09677c83be84\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") "
Sep 09 06:19:25.938469 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:25.938383    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mss76\" (UniqueName: \"kubernetes.io/projected/4a9e76fd-3082-45c8-957f-09677c83be84-kube-api-access-mss76\") pod \"4a9e76fd-3082-45c8-957f-09677c83be84\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") "
Sep 09 06:19:25.938656 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:25.938636    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a9e76fd-3082-45c8-957f-09677c83be84-maas-api-tls\") pod \"4a9e76fd-3082-45c8-957f-09677c83be84\" (UID: \"4a9e76fd-3082-45c8-957f-09677c83be84\") "
Sep 09 06:19:25.939235 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:25.939172    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9e76fd-3082-45c8-957f-09677c83be84-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a9e76fd-3082-45c8-957f-09677c83be84" (UID: "4a9e76fd-3082-45c8-957f-09677c83be84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:19:25.941892 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:25.941865    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a9e76fd-3082-45c8-957f-09677c83be84-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a9e76fd-3082-45c8-957f-09677c83be84" (UID: "4a9e76fd-3082-45c8-957f-09677c83be84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:19:25.942035 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:25.942007    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a9e76fd-3082-45c8-957f-09677c83be84-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4a9e76fd-3082-45c8-957f-09677c83be84" (UID: "4a9e76fd-3082-45c8-957f-09677c83be84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:19:25.942091 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:25.942014    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9e76fd-3082-45c8-957f-09677c83be84-kube-api-access-mss76" (OuterVolumeSpecName: "kube-api-access-mss76") pod "4a9e76fd-3082-45c8-957f-09677c83be84" (UID: "4a9e76fd-3082-45c8-957f-09677c83be84"). InnerVolumeSpecName "kube-api-access-mss76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:19:26.039826 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:26.039737    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a9e76fd-3082-45c8-957f-09677c83be84-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:19:26.039826 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:26.039774    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a9e76fd-3082-45c8-957f-09677c83be84-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:19:26.039826 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:26.039788    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9e76fd-3082-45c8-957f-09677c83be84-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:19:26.039826 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:26.039802    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mss76\" (UniqueName: \"kubernetes.io/projected/4a9e76fd-3082-45c8-957f-09677c83be84-kube-api-access-mss76\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:19:26.139139 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:26.139105    2564 generic.go:356] "Generic (PLEG): container finished" podID="4a9e76fd-3082-45c8-957f-09677c83be84" containerID="df271b1f207e9931fdde7799d8c98eb54a09a920fb4a5646a3cd6b9eef2ce3e5" exitCode=0
Sep 09 06:19:26.139139 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:26.139145    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t" event={"ID":"4a9e76fd-3082-45c8-957f-09677c83be84","Type":"ContainerDied","Data":"df271b1f207e9931fdde7799d8c98eb54a09a920fb4a5646a3cd6b9eef2ce3e5"}
Sep 09 06:19:26.139420 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:26.139161    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"
Sep 09 06:19:26.139420 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:26.139191    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t" event={"ID":"4a9e76fd-3082-45c8-957f-09677c83be84","Type":"ContainerDied","Data":"f4473a0b6807b7c02ba104d1b1fb761a234ffb462a0d3d31bea38b1a1ae0f90f"}
Sep 09 06:19:26.139420 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:26.139213    2564 scope.go:117] "RemoveContainer" containerID="df271b1f207e9931fdde7799d8c98eb54a09a920fb4a5646a3cd6b9eef2ce3e5"
Sep 09 06:19:26.151166 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:26.151144    2564 scope.go:117] "RemoveContainer" containerID="df271b1f207e9931fdde7799d8c98eb54a09a920fb4a5646a3cd6b9eef2ce3e5"
Sep 09 06:19:26.151467 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:19:26.151445    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df271b1f207e9931fdde7799d8c98eb54a09a920fb4a5646a3cd6b9eef2ce3e5\": container with ID starting with df271b1f207e9931fdde7799d8c98eb54a09a920fb4a5646a3cd6b9eef2ce3e5 not found: ID does not exist" containerID="df271b1f207e9931fdde7799d8c98eb54a09a920fb4a5646a3cd6b9eef2ce3e5"
Sep 09 06:19:26.151534 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:26.151478    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df271b1f207e9931fdde7799d8c98eb54a09a920fb4a5646a3cd6b9eef2ce3e5"} err="failed to get container status \"df271b1f207e9931fdde7799d8c98eb54a09a920fb4a5646a3cd6b9eef2ce3e5\": rpc error: code = NotFound desc = could not find container \"df271b1f207e9931fdde7799d8c98eb54a09a920fb4a5646a3cd6b9eef2ce3e5\": container with ID starting with df271b1f207e9931fdde7799d8c98eb54a09a920fb4a5646a3cd6b9eef2ce3e5 not found: ID does not exist"
Sep 09 06:19:26.166838 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:26.166795    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"]
Sep 09 06:19:26.175606 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:26.173142    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a6cf7-646cf84997-pjm2t"]
Sep 09 06:19:27.302930 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:27.302896    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a9e76fd-3082-45c8-957f-09677c83be84" path="/var/lib/kubelet/pods/4a9e76fd-3082-45c8-957f-09677c83be84/volumes"
Sep 09 06:19:33.453860 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:33.453833    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 06:19:33.463894 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:19:33.463869    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 06:20:24.960735 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:24.960692    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"]
Sep 09 06:20:24.961284 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:24.960987    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg" podUID="2520c649-79c2-4776-96a9-0c9534517939" containerName="maas-api" containerID="cri-o://09e6a1e1eba11d7d812f1f7aeb1a1062aab20d97ef5d081eb6a9e26741918e11" gracePeriod=30
Sep 09 06:20:25.218715 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.218653    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:20:25.275189 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.275153    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2520c649-79c2-4776-96a9-0c9534517939-metrics-certs\") pod \"2520c649-79c2-4776-96a9-0c9534517939\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") "
Sep 09 06:20:25.275380 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.275195    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2520c649-79c2-4776-96a9-0c9534517939-maas-api-tls\") pod \"2520c649-79c2-4776-96a9-0c9534517939\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") "
Sep 09 06:20:25.275380 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.275241    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nsjs\" (UniqueName: \"kubernetes.io/projected/2520c649-79c2-4776-96a9-0c9534517939-kube-api-access-5nsjs\") pod \"2520c649-79c2-4776-96a9-0c9534517939\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") "
Sep 09 06:20:25.275380 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.275280    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2520c649-79c2-4776-96a9-0c9534517939-openshift-service-ca\") pod \"2520c649-79c2-4776-96a9-0c9534517939\" (UID: \"2520c649-79c2-4776-96a9-0c9534517939\") "
Sep 09 06:20:25.275745 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.275711    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2520c649-79c2-4776-96a9-0c9534517939-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2520c649-79c2-4776-96a9-0c9534517939" (UID: "2520c649-79c2-4776-96a9-0c9534517939"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:20:25.277227 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.277202    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2520c649-79c2-4776-96a9-0c9534517939-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2520c649-79c2-4776-96a9-0c9534517939" (UID: "2520c649-79c2-4776-96a9-0c9534517939"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:20:25.277338 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.277305    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2520c649-79c2-4776-96a9-0c9534517939-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2520c649-79c2-4776-96a9-0c9534517939" (UID: "2520c649-79c2-4776-96a9-0c9534517939"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:20:25.277386 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.277353    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2520c649-79c2-4776-96a9-0c9534517939-kube-api-access-5nsjs" (OuterVolumeSpecName: "kube-api-access-5nsjs") pod "2520c649-79c2-4776-96a9-0c9534517939" (UID: "2520c649-79c2-4776-96a9-0c9534517939"). InnerVolumeSpecName "kube-api-access-5nsjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:20:25.377193 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.377153    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2520c649-79c2-4776-96a9-0c9534517939-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:20:25.377379 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.377279    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2520c649-79c2-4776-96a9-0c9534517939-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:20:25.377379 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.377303    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nsjs\" (UniqueName: \"kubernetes.io/projected/2520c649-79c2-4776-96a9-0c9534517939-kube-api-access-5nsjs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:20:25.377379 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.377321    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2520c649-79c2-4776-96a9-0c9534517939-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:20:25.493359 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.493268    2564 generic.go:356] "Generic (PLEG): container finished" podID="2520c649-79c2-4776-96a9-0c9534517939" containerID="09e6a1e1eba11d7d812f1f7aeb1a1062aab20d97ef5d081eb6a9e26741918e11" exitCode=0
Sep 09 06:20:25.493359 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.493313    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"
Sep 09 06:20:25.493558 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.493354    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg" event={"ID":"2520c649-79c2-4776-96a9-0c9534517939","Type":"ContainerDied","Data":"09e6a1e1eba11d7d812f1f7aeb1a1062aab20d97ef5d081eb6a9e26741918e11"}
Sep 09 06:20:25.493558 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.493410    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg" event={"ID":"2520c649-79c2-4776-96a9-0c9534517939","Type":"ContainerDied","Data":"4741fe4e1828173b264acdb87caece88a14e9309482dc17f7c963e8b36dee4a0"}
Sep 09 06:20:25.493558 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.493440    2564 scope.go:117] "RemoveContainer" containerID="09e6a1e1eba11d7d812f1f7aeb1a1062aab20d97ef5d081eb6a9e26741918e11"
Sep 09 06:20:25.503380 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.503356    2564 scope.go:117] "RemoveContainer" containerID="09e6a1e1eba11d7d812f1f7aeb1a1062aab20d97ef5d081eb6a9e26741918e11"
Sep 09 06:20:25.503721 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:20:25.503693    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e6a1e1eba11d7d812f1f7aeb1a1062aab20d97ef5d081eb6a9e26741918e11\": container with ID starting with 09e6a1e1eba11d7d812f1f7aeb1a1062aab20d97ef5d081eb6a9e26741918e11 not found: ID does not exist" containerID="09e6a1e1eba11d7d812f1f7aeb1a1062aab20d97ef5d081eb6a9e26741918e11"
Sep 09 06:20:25.503816 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.503725    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e6a1e1eba11d7d812f1f7aeb1a1062aab20d97ef5d081eb6a9e26741918e11"} err="failed to get container status \"09e6a1e1eba11d7d812f1f7aeb1a1062aab20d97ef5d081eb6a9e26741918e11\": rpc error: code = NotFound desc = could not find container \"09e6a1e1eba11d7d812f1f7aeb1a1062aab20d97ef5d081eb6a9e26741918e11\": container with ID starting with 09e6a1e1eba11d7d812f1f7aeb1a1062aab20d97ef5d081eb6a9e26741918e11 not found: ID does not exist"
Sep 09 06:20:25.517309 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.517287    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"]
Sep 09 06:20:25.521987 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:25.521966    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-32ea4d-7fd5fb94f6-r2vdg"]
Sep 09 06:20:27.303285 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:27.303242    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2520c649-79c2-4776-96a9-0c9534517939" path="/var/lib/kubelet/pods/2520c649-79c2-4776-96a9-0c9534517939/volumes"
Sep 09 06:20:42.589666 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:42.589628    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"]
Sep 09 06:20:42.590249 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:42.589887    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl" podUID="d5e9cb12-8d68-45ea-92ed-246f90da8e6b" containerName="maas-api" containerID="cri-o://c4a88e911ea81708b817677aafc40d30df8916b9d6842f4556c1703d4bf4ca18" gracePeriod=30
Sep 09 06:20:42.852550 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:42.852460    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:20:42.948544 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:42.948474    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-maas-api-tls\") pod \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") "
Sep 09 06:20:42.948727 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:42.948581    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-openshift-service-ca\") pod \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") "
Sep 09 06:20:42.948727 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:42.948662    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-metrics-certs\") pod \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") "
Sep 09 06:20:42.948852 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:42.948767    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr2m4\" (UniqueName: \"kubernetes.io/projected/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-kube-api-access-gr2m4\") pod \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\" (UID: \"d5e9cb12-8d68-45ea-92ed-246f90da8e6b\") "
Sep 09 06:20:42.949027 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:42.948971    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5e9cb12-8d68-45ea-92ed-246f90da8e6b" (UID: "d5e9cb12-8d68-45ea-92ed-246f90da8e6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:20:42.949171 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:42.949090    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:20:42.950827 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:42.950803    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d5e9cb12-8d68-45ea-92ed-246f90da8e6b" (UID: "d5e9cb12-8d68-45ea-92ed-246f90da8e6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:20:42.950944 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:42.950927    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-kube-api-access-gr2m4" (OuterVolumeSpecName: "kube-api-access-gr2m4") pod "d5e9cb12-8d68-45ea-92ed-246f90da8e6b" (UID: "d5e9cb12-8d68-45ea-92ed-246f90da8e6b"). InnerVolumeSpecName "kube-api-access-gr2m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:20:42.951113 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:42.951085    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5e9cb12-8d68-45ea-92ed-246f90da8e6b" (UID: "d5e9cb12-8d68-45ea-92ed-246f90da8e6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:20:43.050229 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:43.050197    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:20:43.050229 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:43.050223    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:20:43.050229 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:43.050234    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr2m4\" (UniqueName: \"kubernetes.io/projected/d5e9cb12-8d68-45ea-92ed-246f90da8e6b-kube-api-access-gr2m4\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:20:43.604185 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:43.604154    2564 generic.go:356] "Generic (PLEG): container finished" podID="d5e9cb12-8d68-45ea-92ed-246f90da8e6b" containerID="c4a88e911ea81708b817677aafc40d30df8916b9d6842f4556c1703d4bf4ca18" exitCode=0
Sep 09 06:20:43.604623 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:43.604196    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"
Sep 09 06:20:43.604623 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:43.604240    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl" event={"ID":"d5e9cb12-8d68-45ea-92ed-246f90da8e6b","Type":"ContainerDied","Data":"c4a88e911ea81708b817677aafc40d30df8916b9d6842f4556c1703d4bf4ca18"}
Sep 09 06:20:43.604623 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:43.604291    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl" event={"ID":"d5e9cb12-8d68-45ea-92ed-246f90da8e6b","Type":"ContainerDied","Data":"ccb83c59ff5b4d92297fa19637ad1496eaacc08f9ea60323bba35c827714c3a7"}
Sep 09 06:20:43.604623 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:43.604314    2564 scope.go:117] "RemoveContainer" containerID="c4a88e911ea81708b817677aafc40d30df8916b9d6842f4556c1703d4bf4ca18"
Sep 09 06:20:43.613416 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:43.613400    2564 scope.go:117] "RemoveContainer" containerID="c4a88e911ea81708b817677aafc40d30df8916b9d6842f4556c1703d4bf4ca18"
Sep 09 06:20:43.613714 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:20:43.613692    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a88e911ea81708b817677aafc40d30df8916b9d6842f4556c1703d4bf4ca18\": container with ID starting with c4a88e911ea81708b817677aafc40d30df8916b9d6842f4556c1703d4bf4ca18 not found: ID does not exist" containerID="c4a88e911ea81708b817677aafc40d30df8916b9d6842f4556c1703d4bf4ca18"
Sep 09 06:20:43.613771 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:43.613723    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a88e911ea81708b817677aafc40d30df8916b9d6842f4556c1703d4bf4ca18"} err="failed to get container status \"c4a88e911ea81708b817677aafc40d30df8916b9d6842f4556c1703d4bf4ca18\": rpc error: code = NotFound desc = could not find container \"c4a88e911ea81708b817677aafc40d30df8916b9d6842f4556c1703d4bf4ca18\": container with ID starting with c4a88e911ea81708b817677aafc40d30df8916b9d6842f4556c1703d4bf4ca18 not found: ID does not exist"
Sep 09 06:20:43.620958 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:43.620937    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"]
Sep 09 06:20:43.627059 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:43.627037    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5d89-bbcbc7847-f54bl"]
Sep 09 06:20:45.303141 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:20:45.303109    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5e9cb12-8d68-45ea-92ed-246f90da8e6b" path="/var/lib/kubelet/pods/d5e9cb12-8d68-45ea-92ed-246f90da8e6b/volumes"
Sep 09 06:21:00.108762 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.108562    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"]
Sep 09 06:21:00.109512 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.109434    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7" podUID="5abff106-d69b-418e-a836-cb923ec4cee0" containerName="maas-api" containerID="cri-o://3abaabebcd2a9f4c828a494fbda56080b830d5c5e8c1d9b3a3c5933555d1de61" gracePeriod=30
Sep 09 06:21:00.367465 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.367399    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:21:00.516443 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.516395    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsp9b\" (UniqueName: \"kubernetes.io/projected/5abff106-d69b-418e-a836-cb923ec4cee0-kube-api-access-zsp9b\") pod \"5abff106-d69b-418e-a836-cb923ec4cee0\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") "
Sep 09 06:21:00.516647 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.516458    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abff106-d69b-418e-a836-cb923ec4cee0-openshift-service-ca\") pod \"5abff106-d69b-418e-a836-cb923ec4cee0\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") "
Sep 09 06:21:00.516647 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.516564    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5abff106-d69b-418e-a836-cb923ec4cee0-maas-api-tls\") pod \"5abff106-d69b-418e-a836-cb923ec4cee0\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") "
Sep 09 06:21:00.516647 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.516603    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5abff106-d69b-418e-a836-cb923ec4cee0-metrics-certs\") pod \"5abff106-d69b-418e-a836-cb923ec4cee0\" (UID: \"5abff106-d69b-418e-a836-cb923ec4cee0\") "
Sep 09 06:21:00.516860 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.516835    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abff106-d69b-418e-a836-cb923ec4cee0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5abff106-d69b-418e-a836-cb923ec4cee0" (UID: "5abff106-d69b-418e-a836-cb923ec4cee0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:21:00.518745 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.518715    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abff106-d69b-418e-a836-cb923ec4cee0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5abff106-d69b-418e-a836-cb923ec4cee0" (UID: "5abff106-d69b-418e-a836-cb923ec4cee0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:21:00.518849 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.518745    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abff106-d69b-418e-a836-cb923ec4cee0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5abff106-d69b-418e-a836-cb923ec4cee0" (UID: "5abff106-d69b-418e-a836-cb923ec4cee0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:21:00.518849 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.518796    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abff106-d69b-418e-a836-cb923ec4cee0-kube-api-access-zsp9b" (OuterVolumeSpecName: "kube-api-access-zsp9b") pod "5abff106-d69b-418e-a836-cb923ec4cee0" (UID: "5abff106-d69b-418e-a836-cb923ec4cee0"). InnerVolumeSpecName "kube-api-access-zsp9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:21:00.617297 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.617263    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5abff106-d69b-418e-a836-cb923ec4cee0-maas-api-tls\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:21:00.617297 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.617290    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5abff106-d69b-418e-a836-cb923ec4cee0-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:21:00.617297 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.617300    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsp9b\" (UniqueName: \"kubernetes.io/projected/5abff106-d69b-418e-a836-cb923ec4cee0-kube-api-access-zsp9b\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:21:00.617610 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.617308    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abff106-d69b-418e-a836-cb923ec4cee0-openshift-service-ca\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:21:00.706297 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.706262    2564 generic.go:356] "Generic (PLEG): container finished" podID="5abff106-d69b-418e-a836-cb923ec4cee0" containerID="3abaabebcd2a9f4c828a494fbda56080b830d5c5e8c1d9b3a3c5933555d1de61" exitCode=0
Sep 09 06:21:00.706482 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.706306    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"
Sep 09 06:21:00.706482 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.706349    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7" event={"ID":"5abff106-d69b-418e-a836-cb923ec4cee0","Type":"ContainerDied","Data":"3abaabebcd2a9f4c828a494fbda56080b830d5c5e8c1d9b3a3c5933555d1de61"}
Sep 09 06:21:00.706482 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.706411    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7" event={"ID":"5abff106-d69b-418e-a836-cb923ec4cee0","Type":"ContainerDied","Data":"a035f750c4e1f5b85010a925e192c1f6dc555b4dcb37943ae66fb63c910d07e0"}
Sep 09 06:21:00.706482 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.706442    2564 scope.go:117] "RemoveContainer" containerID="3abaabebcd2a9f4c828a494fbda56080b830d5c5e8c1d9b3a3c5933555d1de61"
Sep 09 06:21:00.717154 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.717136    2564 scope.go:117] "RemoveContainer" containerID="3abaabebcd2a9f4c828a494fbda56080b830d5c5e8c1d9b3a3c5933555d1de61"
Sep 09 06:21:00.717417 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:21:00.717395    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3abaabebcd2a9f4c828a494fbda56080b830d5c5e8c1d9b3a3c5933555d1de61\": container with ID starting with 3abaabebcd2a9f4c828a494fbda56080b830d5c5e8c1d9b3a3c5933555d1de61 not found: ID does not exist" containerID="3abaabebcd2a9f4c828a494fbda56080b830d5c5e8c1d9b3a3c5933555d1de61"
Sep 09 06:21:00.717481 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.717430    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3abaabebcd2a9f4c828a494fbda56080b830d5c5e8c1d9b3a3c5933555d1de61"} err="failed to get container status \"3abaabebcd2a9f4c828a494fbda56080b830d5c5e8c1d9b3a3c5933555d1de61\": rpc error: code = NotFound desc = could not find container \"3abaabebcd2a9f4c828a494fbda56080b830d5c5e8c1d9b3a3c5933555d1de61\": container with ID starting with 3abaabebcd2a9f4c828a494fbda56080b830d5c5e8c1d9b3a3c5933555d1de61 not found: ID does not exist"
Sep 09 06:21:00.732887 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.732859    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"]
Sep 09 06:21:00.737018 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:00.736998    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7"]
Sep 09 06:21:01.303595 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:01.303563    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5abff106-d69b-418e-a836-cb923ec4cee0" path="/var/lib/kubelet/pods/5abff106-d69b-418e-a836-cb923ec4cee0/volumes"
Sep 09 06:21:03.289347 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:03.289303    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cf6be-645d4b67b7-qjsg7" podUID="5abff106-d69b-418e-a836-cb923ec4cee0" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.81:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 06:21:09.668738 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:09.668692    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5945888b8-8rmmg"]
Sep 09 06:21:09.669177 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:09.668914    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5945888b8-8rmmg" podUID="c1087c96-3a96-44df-bfa9-3e155f2b287a" containerName="manager" containerID="cri-o://9381b68ec6c093a272930530e36b6ff8497bf152c03f01f94ef62744f7284e25" gracePeriod=30
Sep 09 06:21:09.916550 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:09.916480    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 06:21:10.007662 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.007572    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97jps\" (UniqueName: \"kubernetes.io/projected/c1087c96-3a96-44df-bfa9-3e155f2b287a-kube-api-access-97jps\") pod \"c1087c96-3a96-44df-bfa9-3e155f2b287a\" (UID: \"c1087c96-3a96-44df-bfa9-3e155f2b287a\") "
Sep 09 06:21:10.007662 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.007628    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1087c96-3a96-44df-bfa9-3e155f2b287a-webhook-cert\") pod \"c1087c96-3a96-44df-bfa9-3e155f2b287a\" (UID: \"c1087c96-3a96-44df-bfa9-3e155f2b287a\") "
Sep 09 06:21:10.007894 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.007684    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1087c96-3a96-44df-bfa9-3e155f2b287a-metrics-certs\") pod \"c1087c96-3a96-44df-bfa9-3e155f2b287a\" (UID: \"c1087c96-3a96-44df-bfa9-3e155f2b287a\") "
Sep 09 06:21:10.009784 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.009753    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1087c96-3a96-44df-bfa9-3e155f2b287a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c1087c96-3a96-44df-bfa9-3e155f2b287a" (UID: "c1087c96-3a96-44df-bfa9-3e155f2b287a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:21:10.009784 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.009769    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1087c96-3a96-44df-bfa9-3e155f2b287a-kube-api-access-97jps" (OuterVolumeSpecName: "kube-api-access-97jps") pod "c1087c96-3a96-44df-bfa9-3e155f2b287a" (UID: "c1087c96-3a96-44df-bfa9-3e155f2b287a"). InnerVolumeSpecName "kube-api-access-97jps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:21:10.009942 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.009781    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1087c96-3a96-44df-bfa9-3e155f2b287a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c1087c96-3a96-44df-bfa9-3e155f2b287a" (UID: "c1087c96-3a96-44df-bfa9-3e155f2b287a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:21:10.108933 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.108888    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1087c96-3a96-44df-bfa9-3e155f2b287a-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:21:10.108933 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.108924    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97jps\" (UniqueName: \"kubernetes.io/projected/c1087c96-3a96-44df-bfa9-3e155f2b287a-kube-api-access-97jps\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:21:10.108933 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.108939    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1087c96-3a96-44df-bfa9-3e155f2b287a-webhook-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:21:10.767171 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.767138    2564 generic.go:356] "Generic (PLEG): container finished" podID="c1087c96-3a96-44df-bfa9-3e155f2b287a" containerID="9381b68ec6c093a272930530e36b6ff8497bf152c03f01f94ef62744f7284e25" exitCode=0
Sep 09 06:21:10.767582 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.767183    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5945888b8-8rmmg"
Sep 09 06:21:10.767582 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.767217    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5945888b8-8rmmg" event={"ID":"c1087c96-3a96-44df-bfa9-3e155f2b287a","Type":"ContainerDied","Data":"9381b68ec6c093a272930530e36b6ff8497bf152c03f01f94ef62744f7284e25"}
Sep 09 06:21:10.767582 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.767267    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5945888b8-8rmmg" event={"ID":"c1087c96-3a96-44df-bfa9-3e155f2b287a","Type":"ContainerDied","Data":"d172e8d6e5b1ededd4543743cc870b9294417c8fb8ec4bd0458e230f6aaa0990"}
Sep 09 06:21:10.767582 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.767295    2564 scope.go:117] "RemoveContainer" containerID="9381b68ec6c093a272930530e36b6ff8497bf152c03f01f94ef62744f7284e25"
Sep 09 06:21:10.777593 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.777573    2564 scope.go:117] "RemoveContainer" containerID="9381b68ec6c093a272930530e36b6ff8497bf152c03f01f94ef62744f7284e25"
Sep 09 06:21:10.777833 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:21:10.777817    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9381b68ec6c093a272930530e36b6ff8497bf152c03f01f94ef62744f7284e25\": container with ID starting with 9381b68ec6c093a272930530e36b6ff8497bf152c03f01f94ef62744f7284e25 not found: ID does not exist" containerID="9381b68ec6c093a272930530e36b6ff8497bf152c03f01f94ef62744f7284e25"
Sep 09 06:21:10.777882 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.777841    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9381b68ec6c093a272930530e36b6ff8497bf152c03f01f94ef62744f7284e25"} err="failed to get container status \"9381b68ec6c093a272930530e36b6ff8497bf152c03f01f94ef62744f7284e25\": rpc error: code = NotFound desc = could not find container \"9381b68ec6c093a272930530e36b6ff8497bf152c03f01f94ef62744f7284e25\": container with ID starting with 9381b68ec6c093a272930530e36b6ff8497bf152c03f01f94ef62744f7284e25 not found: ID does not exist"
Sep 09 06:21:10.791137 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.791104    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5945888b8-8rmmg"]
Sep 09 06:21:10.795211 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:10.795181    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5945888b8-8rmmg"]
Sep 09 06:21:11.302610 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:21:11.302571    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1087c96-3a96-44df-bfa9-3e155f2b287a" path="/var/lib/kubelet/pods/c1087c96-3a96-44df-bfa9-3e155f2b287a/volumes"
Sep 09 06:22:11.594963 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.594924    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5945888b8-qktn2"]
Sep 09 06:22:11.595472 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595426    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1087c96-3a96-44df-bfa9-3e155f2b287a" containerName="manager"
Sep 09 06:22:11.595472 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595443    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1087c96-3a96-44df-bfa9-3e155f2b287a" containerName="manager"
Sep 09 06:22:11.595472 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595451    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5abff106-d69b-418e-a836-cb923ec4cee0" containerName="maas-api"
Sep 09 06:22:11.595472 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595457    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5abff106-d69b-418e-a836-cb923ec4cee0" containerName="maas-api"
Sep 09 06:22:11.595472 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595474    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9e76fd-3082-45c8-957f-09677c83be84" containerName="maas-api"
Sep 09 06:22:11.595686 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595481    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9e76fd-3082-45c8-957f-09677c83be84" containerName="maas-api"
Sep 09 06:22:11.595686 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595487    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2520c649-79c2-4776-96a9-0c9534517939" containerName="maas-api"
Sep 09 06:22:11.595686 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595506    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2520c649-79c2-4776-96a9-0c9534517939" containerName="maas-api"
Sep 09 06:22:11.595686 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595514    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5e9cb12-8d68-45ea-92ed-246f90da8e6b" containerName="maas-api"
Sep 09 06:22:11.595686 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595520    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5e9cb12-8d68-45ea-92ed-246f90da8e6b" containerName="maas-api"
Sep 09 06:22:11.595686 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595535    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1751498-77bf-4c05-bb4c-78a123946ec4" containerName="maas-api"
Sep 09 06:22:11.595686 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595540    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1751498-77bf-4c05-bb4c-78a123946ec4" containerName="maas-api"
Sep 09 06:22:11.595686 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595598    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1087c96-3a96-44df-bfa9-3e155f2b287a" containerName="manager"
Sep 09 06:22:11.595686 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595606    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5abff106-d69b-418e-a836-cb923ec4cee0" containerName="maas-api"
Sep 09 06:22:11.595686 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595617    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a9e76fd-3082-45c8-957f-09677c83be84" containerName="maas-api"
Sep 09 06:22:11.595686 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595627    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5e9cb12-8d68-45ea-92ed-246f90da8e6b" containerName="maas-api"
Sep 09 06:22:11.595686 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595634    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1751498-77bf-4c05-bb4c-78a123946ec4" containerName="maas-api"
Sep 09 06:22:11.595686 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.595641    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2520c649-79c2-4776-96a9-0c9534517939" containerName="maas-api"
Sep 09 06:22:11.599255 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.599151    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5945888b8-qktn2"
Sep 09 06:22:11.607204 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.607179    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-b4428\""
Sep 09 06:22:11.607585 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.607556    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 06:22:11.607715 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.607630    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 06:22:11.607715 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.607558    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 06:22:11.610480 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.610456    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5945888b8-qktn2"]
Sep 09 06:22:11.669114 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.669082    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77fjb\" (UniqueName: \"kubernetes.io/projected/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-kube-api-access-77fjb\") pod \"maas-controller-5945888b8-qktn2\" (UID: \"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4\") " pod="opendatahub/maas-controller-5945888b8-qktn2"
Sep 09 06:22:11.669268 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.669179    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-metrics-certs\") pod \"maas-controller-5945888b8-qktn2\" (UID: \"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4\") " pod="opendatahub/maas-controller-5945888b8-qktn2"
Sep 09 06:22:11.669268 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.669201    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-webhook-cert\") pod \"maas-controller-5945888b8-qktn2\" (UID: \"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4\") " pod="opendatahub/maas-controller-5945888b8-qktn2"
Sep 09 06:22:11.770337 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.770304    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-metrics-certs\") pod \"maas-controller-5945888b8-qktn2\" (UID: \"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4\") " pod="opendatahub/maas-controller-5945888b8-qktn2"
Sep 09 06:22:11.770337 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.770340    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-webhook-cert\") pod \"maas-controller-5945888b8-qktn2\" (UID: \"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4\") " pod="opendatahub/maas-controller-5945888b8-qktn2"
Sep 09 06:22:11.770571 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.770426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77fjb\" (UniqueName: \"kubernetes.io/projected/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-kube-api-access-77fjb\") pod \"maas-controller-5945888b8-qktn2\" (UID: \"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4\") " pod="opendatahub/maas-controller-5945888b8-qktn2"
Sep 09 06:22:11.772783 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.772759    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-metrics-certs\") pod \"maas-controller-5945888b8-qktn2\" (UID: \"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4\") " pod="opendatahub/maas-controller-5945888b8-qktn2"
Sep 09 06:22:11.772914 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.772892    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-webhook-cert\") pod \"maas-controller-5945888b8-qktn2\" (UID: \"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4\") " pod="opendatahub/maas-controller-5945888b8-qktn2"
Sep 09 06:22:11.779177 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.779155    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77fjb\" (UniqueName: \"kubernetes.io/projected/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-kube-api-access-77fjb\") pod \"maas-controller-5945888b8-qktn2\" (UID: \"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4\") " pod="opendatahub/maas-controller-5945888b8-qktn2"
Sep 09 06:22:11.916430 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:11.916345    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5945888b8-qktn2"
Sep 09 06:22:12.190485 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:12.190457    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5945888b8-qktn2"]
Sep 09 06:22:12.192102 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:22:12.192075    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd638d1_94f8_4c1f_a4f3_441e23cf75b4.slice/crio-3c2182488493c826bd72a69e943f7f370dcf7ec5edf417f1b4bead5dfca09315 WatchSource:0}: Error finding container 3c2182488493c826bd72a69e943f7f370dcf7ec5edf417f1b4bead5dfca09315: Status 404 returned error can't find the container with id 3c2182488493c826bd72a69e943f7f370dcf7ec5edf417f1b4bead5dfca09315
Sep 09 06:22:13.116654 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:13.116613    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5945888b8-qktn2" event={"ID":"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4","Type":"ContainerStarted","Data":"35554367ccbf5b88bbdcf9ffaa6fe8d1bdf7d3c6e9b337b391fdcf79e610b6c0"}
Sep 09 06:22:13.117027 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:13.116679    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5945888b8-qktn2" event={"ID":"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4","Type":"ContainerStarted","Data":"3c2182488493c826bd72a69e943f7f370dcf7ec5edf417f1b4bead5dfca09315"}
Sep 09 06:22:13.117027 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:13.116826    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5945888b8-qktn2"
Sep 09 06:22:13.136291 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:13.136248    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5945888b8-qktn2" podStartSLOduration=1.53684734 podStartE2EDuration="2.136234927s" totalImagesPullingTime="599.387587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:22:13.133633923 +0000 UTC m=+1960.349963807" watchObservedRunningTime="2026-09-09 06:22:13.136234927 +0000 UTC m=+1960.352564836"
Sep 09 06:22:24.125337 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:24.125306    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5945888b8-qktn2"
Sep 09 06:22:33.486553 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:33.486463    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5945888b8-qktn2"]
Sep 09 06:22:33.487007 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:33.486709    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5945888b8-qktn2" podUID="ecd638d1-94f8-4c1f-a4f3-441e23cf75b4" containerName="manager" containerID="cri-o://35554367ccbf5b88bbdcf9ffaa6fe8d1bdf7d3c6e9b337b391fdcf79e610b6c0" gracePeriod=30
Sep 09 06:22:33.735467 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:33.735443    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5945888b8-qktn2"
Sep 09 06:22:33.877366 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:33.877334    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-webhook-cert\") pod \"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4\" (UID: \"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4\") "
Sep 09 06:22:33.877556 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:33.877436    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77fjb\" (UniqueName: \"kubernetes.io/projected/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-kube-api-access-77fjb\") pod \"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4\" (UID: \"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4\") "
Sep 09 06:22:33.877556 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:33.877531    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-metrics-certs\") pod \"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4\" (UID: \"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4\") "
Sep 09 06:22:33.879481 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:33.879449    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-kube-api-access-77fjb" (OuterVolumeSpecName: "kube-api-access-77fjb") pod "ecd638d1-94f8-4c1f-a4f3-441e23cf75b4" (UID: "ecd638d1-94f8-4c1f-a4f3-441e23cf75b4"). InnerVolumeSpecName "kube-api-access-77fjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:22:33.879649 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:33.879510    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ecd638d1-94f8-4c1f-a4f3-441e23cf75b4" (UID: "ecd638d1-94f8-4c1f-a4f3-441e23cf75b4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:22:33.879649 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:33.879534    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ecd638d1-94f8-4c1f-a4f3-441e23cf75b4" (UID: "ecd638d1-94f8-4c1f-a4f3-441e23cf75b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:22:33.978810 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:33.978780    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77fjb\" (UniqueName: \"kubernetes.io/projected/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-kube-api-access-77fjb\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:22:33.978810 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:33.978806    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-metrics-certs\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:22:33.978810 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:33.978816    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4-webhook-cert\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:22:34.250829 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:34.250739    2564 generic.go:356] "Generic (PLEG): container finished" podID="ecd638d1-94f8-4c1f-a4f3-441e23cf75b4" containerID="35554367ccbf5b88bbdcf9ffaa6fe8d1bdf7d3c6e9b337b391fdcf79e610b6c0" exitCode=0
Sep 09 06:22:34.250829 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:34.250793    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5945888b8-qktn2"
Sep 09 06:22:34.251071 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:34.250826    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5945888b8-qktn2" event={"ID":"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4","Type":"ContainerDied","Data":"35554367ccbf5b88bbdcf9ffaa6fe8d1bdf7d3c6e9b337b391fdcf79e610b6c0"}
Sep 09 06:22:34.251071 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:34.250890    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5945888b8-qktn2" event={"ID":"ecd638d1-94f8-4c1f-a4f3-441e23cf75b4","Type":"ContainerDied","Data":"3c2182488493c826bd72a69e943f7f370dcf7ec5edf417f1b4bead5dfca09315"}
Sep 09 06:22:34.251071 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:34.250919    2564 scope.go:117] "RemoveContainer" containerID="35554367ccbf5b88bbdcf9ffaa6fe8d1bdf7d3c6e9b337b391fdcf79e610b6c0"
Sep 09 06:22:34.260761 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:34.260743    2564 scope.go:117] "RemoveContainer" containerID="35554367ccbf5b88bbdcf9ffaa6fe8d1bdf7d3c6e9b337b391fdcf79e610b6c0"
Sep 09 06:22:34.260999 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:22:34.260981    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35554367ccbf5b88bbdcf9ffaa6fe8d1bdf7d3c6e9b337b391fdcf79e610b6c0\": container with ID starting with 35554367ccbf5b88bbdcf9ffaa6fe8d1bdf7d3c6e9b337b391fdcf79e610b6c0 not found: ID does not exist" containerID="35554367ccbf5b88bbdcf9ffaa6fe8d1bdf7d3c6e9b337b391fdcf79e610b6c0"
Sep 09 06:22:34.261051 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:34.261010    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35554367ccbf5b88bbdcf9ffaa6fe8d1bdf7d3c6e9b337b391fdcf79e610b6c0"} err="failed to get container status \"35554367ccbf5b88bbdcf9ffaa6fe8d1bdf7d3c6e9b337b391fdcf79e610b6c0\": rpc error: code = NotFound desc = could not find container \"35554367ccbf5b88bbdcf9ffaa6fe8d1bdf7d3c6e9b337b391fdcf79e610b6c0\": container with ID starting with 35554367ccbf5b88bbdcf9ffaa6fe8d1bdf7d3c6e9b337b391fdcf79e610b6c0 not found: ID does not exist"
Sep 09 06:22:34.274008 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:34.273987    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5945888b8-qktn2"]
Sep 09 06:22:34.277947 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:34.277921    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5945888b8-qktn2"]
Sep 09 06:22:35.065096 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.065063    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5945888b8-vv4kz"]
Sep 09 06:22:35.065522 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.065508    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecd638d1-94f8-4c1f-a4f3-441e23cf75b4" containerName="manager"
Sep 09 06:22:35.065522 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.065523    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecd638d1-94f8-4c1f-a4f3-441e23cf75b4" containerName="manager"
Sep 09 06:22:35.065602 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.065597    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecd638d1-94f8-4c1f-a4f3-441e23cf75b4" containerName="manager"
Sep 09 06:22:35.069987 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.069968    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5945888b8-vv4kz"
Sep 09 06:22:35.073664 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.073614    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 06:22:35.073664 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.073657    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 06:22:35.073853 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.073715    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-b4428\""
Sep 09 06:22:35.074130 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.074102    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 06:22:35.078653 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.078292    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5945888b8-vv4kz"]
Sep 09 06:22:35.190129 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.190092    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c8fb26-6722-49a2-88a3-39d52db0c0c2-metrics-certs\") pod \"maas-controller-5945888b8-vv4kz\" (UID: \"95c8fb26-6722-49a2-88a3-39d52db0c0c2\") " pod="opendatahub/maas-controller-5945888b8-vv4kz"
Sep 09 06:22:35.190275 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.190255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p52jf\" (UniqueName: \"kubernetes.io/projected/95c8fb26-6722-49a2-88a3-39d52db0c0c2-kube-api-access-p52jf\") pod \"maas-controller-5945888b8-vv4kz\" (UID: \"95c8fb26-6722-49a2-88a3-39d52db0c0c2\") " pod="opendatahub/maas-controller-5945888b8-vv4kz"
Sep 09 06:22:35.190338 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.190286    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95c8fb26-6722-49a2-88a3-39d52db0c0c2-webhook-cert\") pod \"maas-controller-5945888b8-vv4kz\" (UID: \"95c8fb26-6722-49a2-88a3-39d52db0c0c2\") " pod="opendatahub/maas-controller-5945888b8-vv4kz"
Sep 09 06:22:35.290946 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.290909    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p52jf\" (UniqueName: \"kubernetes.io/projected/95c8fb26-6722-49a2-88a3-39d52db0c0c2-kube-api-access-p52jf\") pod \"maas-controller-5945888b8-vv4kz\" (UID: \"95c8fb26-6722-49a2-88a3-39d52db0c0c2\") " pod="opendatahub/maas-controller-5945888b8-vv4kz"
Sep 09 06:22:35.290946 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.290950    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95c8fb26-6722-49a2-88a3-39d52db0c0c2-webhook-cert\") pod \"maas-controller-5945888b8-vv4kz\" (UID: \"95c8fb26-6722-49a2-88a3-39d52db0c0c2\") " pod="opendatahub/maas-controller-5945888b8-vv4kz"
Sep 09 06:22:35.291155 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.291062    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c8fb26-6722-49a2-88a3-39d52db0c0c2-metrics-certs\") pod \"maas-controller-5945888b8-vv4kz\" (UID: \"95c8fb26-6722-49a2-88a3-39d52db0c0c2\") " pod="opendatahub/maas-controller-5945888b8-vv4kz"
Sep 09 06:22:35.293544 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.293513    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c8fb26-6722-49a2-88a3-39d52db0c0c2-metrics-certs\") pod \"maas-controller-5945888b8-vv4kz\" (UID: \"95c8fb26-6722-49a2-88a3-39d52db0c0c2\") " pod="opendatahub/maas-controller-5945888b8-vv4kz"
Sep 09 06:22:35.293665 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.293637    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95c8fb26-6722-49a2-88a3-39d52db0c0c2-webhook-cert\") pod \"maas-controller-5945888b8-vv4kz\" (UID: \"95c8fb26-6722-49a2-88a3-39d52db0c0c2\") " pod="opendatahub/maas-controller-5945888b8-vv4kz"
Sep 09 06:22:35.299695 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.299677    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p52jf\" (UniqueName: \"kubernetes.io/projected/95c8fb26-6722-49a2-88a3-39d52db0c0c2-kube-api-access-p52jf\") pod \"maas-controller-5945888b8-vv4kz\" (UID: \"95c8fb26-6722-49a2-88a3-39d52db0c0c2\") " pod="opendatahub/maas-controller-5945888b8-vv4kz"
Sep 09 06:22:35.303914 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.303890    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecd638d1-94f8-4c1f-a4f3-441e23cf75b4" path="/var/lib/kubelet/pods/ecd638d1-94f8-4c1f-a4f3-441e23cf75b4/volumes"
Sep 09 06:22:35.382894 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.382865    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5945888b8-vv4kz"
Sep 09 06:22:35.655103 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.655079    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5945888b8-vv4kz"]
Sep 09 06:22:35.656815 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:22:35.656777    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c8fb26_6722_49a2_88a3_39d52db0c0c2.slice/crio-cb935b3bd3309dd2f6c6ece3c49e6a04f15dae0e9d4621612a86730960fa1175 WatchSource:0}: Error finding container cb935b3bd3309dd2f6c6ece3c49e6a04f15dae0e9d4621612a86730960fa1175: Status 404 returned error can't find the container with id cb935b3bd3309dd2f6c6ece3c49e6a04f15dae0e9d4621612a86730960fa1175
Sep 09 06:22:35.658266 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:35.658243    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 06:22:36.265292 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:36.265254    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5945888b8-vv4kz" event={"ID":"95c8fb26-6722-49a2-88a3-39d52db0c0c2","Type":"ContainerStarted","Data":"aef58cc5a32b863cf9cfa1a0d24d6e34fedff45e46ee568c16064f56b2718c96"}
Sep 09 06:22:36.265758 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:36.265302    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5945888b8-vv4kz" event={"ID":"95c8fb26-6722-49a2-88a3-39d52db0c0c2","Type":"ContainerStarted","Data":"cb935b3bd3309dd2f6c6ece3c49e6a04f15dae0e9d4621612a86730960fa1175"}
Sep 09 06:22:36.265758 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:36.265358    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5945888b8-vv4kz"
Sep 09 06:22:36.283598 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:36.283548    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5945888b8-vv4kz" podStartSLOduration=0.880208085 podStartE2EDuration="1.28353392s" totalImagesPullingTime="403.325835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:22:36.282241103 +0000 UTC m=+1983.498570989" watchObservedRunningTime="2026-09-09 06:22:36.28353392 +0000 UTC m=+1983.499863804"
Sep 09 06:22:47.273857 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:22:47.273821    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5945888b8-vv4kz"
Sep 09 06:23:51.018519 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.018476    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"]
Sep 09 06:23:51.018956 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.018674    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb" podUID="353bec02-f547-4b97-a2f6-a0b8d5f4b3fd" containerName="manager" containerID="cri-o://a1f41c414ab54a2fa327d7861395c3d793f62c73808d0c4bc759ba18fb34d0a6" gracePeriod=10
Sep 09 06:23:51.275377 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.275321    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"
Sep 09 06:23:51.391636 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.391593    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/353bec02-f547-4b97-a2f6-a0b8d5f4b3fd-extensions-socket-volume\") pod \"353bec02-f547-4b97-a2f6-a0b8d5f4b3fd\" (UID: \"353bec02-f547-4b97-a2f6-a0b8d5f4b3fd\") "
Sep 09 06:23:51.391794 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.391693    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h7fv\" (UniqueName: \"kubernetes.io/projected/353bec02-f547-4b97-a2f6-a0b8d5f4b3fd-kube-api-access-6h7fv\") pod \"353bec02-f547-4b97-a2f6-a0b8d5f4b3fd\" (UID: \"353bec02-f547-4b97-a2f6-a0b8d5f4b3fd\") "
Sep 09 06:23:51.391976 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.391950    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353bec02-f547-4b97-a2f6-a0b8d5f4b3fd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "353bec02-f547-4b97-a2f6-a0b8d5f4b3fd" (UID: "353bec02-f547-4b97-a2f6-a0b8d5f4b3fd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:23:51.393720 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.393695    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353bec02-f547-4b97-a2f6-a0b8d5f4b3fd-kube-api-access-6h7fv" (OuterVolumeSpecName: "kube-api-access-6h7fv") pod "353bec02-f547-4b97-a2f6-a0b8d5f4b3fd" (UID: "353bec02-f547-4b97-a2f6-a0b8d5f4b3fd"). InnerVolumeSpecName "kube-api-access-6h7fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:23:51.492754 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.492717    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/353bec02-f547-4b97-a2f6-a0b8d5f4b3fd-extensions-socket-volume\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:23:51.492754 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.492748    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h7fv\" (UniqueName: \"kubernetes.io/projected/353bec02-f547-4b97-a2f6-a0b8d5f4b3fd-kube-api-access-6h7fv\") on node \"ip-10-0-131-250.ec2.internal\" DevicePath \"\""
Sep 09 06:23:51.712060 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.712026    2564 generic.go:356] "Generic (PLEG): container finished" podID="353bec02-f547-4b97-a2f6-a0b8d5f4b3fd" containerID="a1f41c414ab54a2fa327d7861395c3d793f62c73808d0c4bc759ba18fb34d0a6" exitCode=0
Sep 09 06:23:51.712262 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.712074    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"
Sep 09 06:23:51.712262 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.712108    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb" event={"ID":"353bec02-f547-4b97-a2f6-a0b8d5f4b3fd","Type":"ContainerDied","Data":"a1f41c414ab54a2fa327d7861395c3d793f62c73808d0c4bc759ba18fb34d0a6"}
Sep 09 06:23:51.712262 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.712167    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb" event={"ID":"353bec02-f547-4b97-a2f6-a0b8d5f4b3fd","Type":"ContainerDied","Data":"1d0117e3f9f8481df17b2ebbaf71ec78ff55ccaff1f4712301b9b43e2ab63b81"}
Sep 09 06:23:51.712262 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.712195    2564 scope.go:117] "RemoveContainer" containerID="a1f41c414ab54a2fa327d7861395c3d793f62c73808d0c4bc759ba18fb34d0a6"
Sep 09 06:23:51.721971 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.721956    2564 scope.go:117] "RemoveContainer" containerID="a1f41c414ab54a2fa327d7861395c3d793f62c73808d0c4bc759ba18fb34d0a6"
Sep 09 06:23:51.722258 ip-10-0-131-250 kubenswrapper[2564]: E0909 06:23:51.722237    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f41c414ab54a2fa327d7861395c3d793f62c73808d0c4bc759ba18fb34d0a6\": container with ID starting with a1f41c414ab54a2fa327d7861395c3d793f62c73808d0c4bc759ba18fb34d0a6 not found: ID does not exist" containerID="a1f41c414ab54a2fa327d7861395c3d793f62c73808d0c4bc759ba18fb34d0a6"
Sep 09 06:23:51.722330 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.722265    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f41c414ab54a2fa327d7861395c3d793f62c73808d0c4bc759ba18fb34d0a6"} err="failed to get container status \"a1f41c414ab54a2fa327d7861395c3d793f62c73808d0c4bc759ba18fb34d0a6\": rpc error: code = NotFound desc = could not find container \"a1f41c414ab54a2fa327d7861395c3d793f62c73808d0c4bc759ba18fb34d0a6\": container with ID starting with a1f41c414ab54a2fa327d7861395c3d793f62c73808d0c4bc759ba18fb34d0a6 not found: ID does not exist"
Sep 09 06:23:51.735647 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.735622    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"]
Sep 09 06:23:51.741400 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:51.741381    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wsnzb"]
Sep 09 06:23:53.302927 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:23:53.302894    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="353bec02-f547-4b97-a2f6-a0b8d5f4b3fd" path="/var/lib/kubelet/pods/353bec02-f547-4b97-a2f6-a0b8d5f4b3fd/volumes"
Sep 09 06:24:33.494121 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:33.494091    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 06:24:33.503968 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:33.503946    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 06:24:51.745744 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:51.745693    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c"]
Sep 09 06:24:51.746220 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:51.746106    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="353bec02-f547-4b97-a2f6-a0b8d5f4b3fd" containerName="manager"
Sep 09 06:24:51.746220 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:51.746118    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="353bec02-f547-4b97-a2f6-a0b8d5f4b3fd" containerName="manager"
Sep 09 06:24:51.746220 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:51.746191    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="353bec02-f547-4b97-a2f6-a0b8d5f4b3fd" containerName="manager"
Sep 09 06:24:51.749145 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:51.749128    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c"
Sep 09 06:24:51.751864 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:51.751843    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pm7rx\""
Sep 09 06:24:51.764925 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:51.764901    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c"]
Sep 09 06:24:51.861085 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:51.861047    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d47458d2-76f6-4db9-a2e8-8dec382bbf8e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-85w5c\" (UID: \"d47458d2-76f6-4db9-a2e8-8dec382bbf8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c"
Sep 09 06:24:51.861265 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:51.861110    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbcdr\" (UniqueName: \"kubernetes.io/projected/d47458d2-76f6-4db9-a2e8-8dec382bbf8e-kube-api-access-nbcdr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-85w5c\" (UID: \"d47458d2-76f6-4db9-a2e8-8dec382bbf8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c"
Sep 09 06:24:51.962295 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:51.962257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d47458d2-76f6-4db9-a2e8-8dec382bbf8e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-85w5c\" (UID: \"d47458d2-76f6-4db9-a2e8-8dec382bbf8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c"
Sep 09 06:24:51.962476 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:51.962330    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbcdr\" (UniqueName: \"kubernetes.io/projected/d47458d2-76f6-4db9-a2e8-8dec382bbf8e-kube-api-access-nbcdr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-85w5c\" (UID: \"d47458d2-76f6-4db9-a2e8-8dec382bbf8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c"
Sep 09 06:24:51.962735 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:51.962711    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d47458d2-76f6-4db9-a2e8-8dec382bbf8e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-85w5c\" (UID: \"d47458d2-76f6-4db9-a2e8-8dec382bbf8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c"
Sep 09 06:24:51.973245 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:51.973219    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbcdr\" (UniqueName: \"kubernetes.io/projected/d47458d2-76f6-4db9-a2e8-8dec382bbf8e-kube-api-access-nbcdr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-85w5c\" (UID: \"d47458d2-76f6-4db9-a2e8-8dec382bbf8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c"
Sep 09 06:24:52.059688 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:52.059590    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c"
Sep 09 06:24:52.345900 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:52.345868    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c"]
Sep 09 06:24:52.348819 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:24:52.348789    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47458d2_76f6_4db9_a2e8_8dec382bbf8e.slice/crio-eb57d55dce9820ea1fc2f702c67282aee518fdf928421d3c46ebbaecf111470c WatchSource:0}: Error finding container eb57d55dce9820ea1fc2f702c67282aee518fdf928421d3c46ebbaecf111470c: Status 404 returned error can't find the container with id eb57d55dce9820ea1fc2f702c67282aee518fdf928421d3c46ebbaecf111470c
Sep 09 06:24:53.068112 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:53.068075    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c" event={"ID":"d47458d2-76f6-4db9-a2e8-8dec382bbf8e","Type":"ContainerStarted","Data":"8ee12711c65bb685468e08ffe7ea1bcb383420f650089920832dd850ab27fa07"}
Sep 09 06:24:53.068601 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:53.068131    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c" event={"ID":"d47458d2-76f6-4db9-a2e8-8dec382bbf8e","Type":"ContainerStarted","Data":"eb57d55dce9820ea1fc2f702c67282aee518fdf928421d3c46ebbaecf111470c"}
Sep 09 06:24:53.068601 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:53.068258    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c"
Sep 09 06:24:53.097261 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:24:53.097215    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c" podStartSLOduration=2.097200713 podStartE2EDuration="2.097200713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:24:53.09369931 +0000 UTC m=+2120.310029194" watchObservedRunningTime="2026-09-09 06:24:53.097200713 +0000 UTC m=+2120.313530597"
Sep 09 06:25:04.077510 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:25:04.077464    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-85w5c"
Sep 09 06:26:33.102287 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.102249    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"]
Sep 09 06:26:33.106145 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.106122    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:33.109964 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.109945    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 06:26:33.110090 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.109994    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 06:26:33.110090 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.109962    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-927tl\""
Sep 09 06:26:33.110090 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.109955    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 06:26:33.110090 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.110032    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 06:26:33.115292 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.115271    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"]
Sep 09 06:26:33.191775 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.191741    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c74b318-d35d-4166-ade9-c56e4079a2e6-maas-api-tls\") pod \"maas-api-6489fd66cd-7xv5d\" (UID: \"9c74b318-d35d-4166-ade9-c56e4079a2e6\") " pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:33.191951 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.191784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlg8d\" (UniqueName: \"kubernetes.io/projected/9c74b318-d35d-4166-ade9-c56e4079a2e6-kube-api-access-vlg8d\") pod \"maas-api-6489fd66cd-7xv5d\" (UID: \"9c74b318-d35d-4166-ade9-c56e4079a2e6\") " pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:33.191951 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.191807    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c74b318-d35d-4166-ade9-c56e4079a2e6-metrics-certs\") pod \"maas-api-6489fd66cd-7xv5d\" (UID: \"9c74b318-d35d-4166-ade9-c56e4079a2e6\") " pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:33.191951 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.191875    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c74b318-d35d-4166-ade9-c56e4079a2e6-openshift-service-ca\") pod \"maas-api-6489fd66cd-7xv5d\" (UID: \"9c74b318-d35d-4166-ade9-c56e4079a2e6\") " pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:33.293188 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.293153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c74b318-d35d-4166-ade9-c56e4079a2e6-maas-api-tls\") pod \"maas-api-6489fd66cd-7xv5d\" (UID: \"9c74b318-d35d-4166-ade9-c56e4079a2e6\") " pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:33.293367 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.293206    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlg8d\" (UniqueName: \"kubernetes.io/projected/9c74b318-d35d-4166-ade9-c56e4079a2e6-kube-api-access-vlg8d\") pod \"maas-api-6489fd66cd-7xv5d\" (UID: \"9c74b318-d35d-4166-ade9-c56e4079a2e6\") " pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:33.293367 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.293233    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c74b318-d35d-4166-ade9-c56e4079a2e6-metrics-certs\") pod \"maas-api-6489fd66cd-7xv5d\" (UID: \"9c74b318-d35d-4166-ade9-c56e4079a2e6\") " pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:33.293367 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.293303    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c74b318-d35d-4166-ade9-c56e4079a2e6-openshift-service-ca\") pod \"maas-api-6489fd66cd-7xv5d\" (UID: \"9c74b318-d35d-4166-ade9-c56e4079a2e6\") " pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:33.295934 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.295906    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 06:26:33.296060 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.296027    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 06:26:33.296114 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.296065    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 06:26:33.301349 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.301326    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 06:26:33.304428 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.304400    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c74b318-d35d-4166-ade9-c56e4079a2e6-openshift-service-ca\") pod \"maas-api-6489fd66cd-7xv5d\" (UID: \"9c74b318-d35d-4166-ade9-c56e4079a2e6\") " pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:33.305991 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.305969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c74b318-d35d-4166-ade9-c56e4079a2e6-metrics-certs\") pod \"maas-api-6489fd66cd-7xv5d\" (UID: \"9c74b318-d35d-4166-ade9-c56e4079a2e6\") " pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:33.306125 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.306109    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c74b318-d35d-4166-ade9-c56e4079a2e6-maas-api-tls\") pod \"maas-api-6489fd66cd-7xv5d\" (UID: \"9c74b318-d35d-4166-ade9-c56e4079a2e6\") " pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:33.311916 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.311894    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlg8d\" (UniqueName: \"kubernetes.io/projected/9c74b318-d35d-4166-ade9-c56e4079a2e6-kube-api-access-vlg8d\") pod \"maas-api-6489fd66cd-7xv5d\" (UID: \"9c74b318-d35d-4166-ade9-c56e4079a2e6\") " pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:33.420376 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.420301    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-927tl\""
Sep 09 06:26:33.428196 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.428177    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:33.701384 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:33.701353    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"]
Sep 09 06:26:33.702898 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:26:33.702872    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c74b318_d35d_4166_ade9_c56e4079a2e6.slice/crio-15da42bfe77b2ac16e5f976ef96e799820f4ce21980d23af07998d460476e871 WatchSource:0}: Error finding container 15da42bfe77b2ac16e5f976ef96e799820f4ce21980d23af07998d460476e871: Status 404 returned error can't find the container with id 15da42bfe77b2ac16e5f976ef96e799820f4ce21980d23af07998d460476e871
Sep 09 06:26:34.665768 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:34.665731    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d" event={"ID":"9c74b318-d35d-4166-ade9-c56e4079a2e6","Type":"ContainerStarted","Data":"c776c3e75390416b0d32bf9e430a41c46d4ba59e2e38cd0b0571b05789f7f44f"}
Sep 09 06:26:34.666169 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:34.665786    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d" event={"ID":"9c74b318-d35d-4166-ade9-c56e4079a2e6","Type":"ContainerStarted","Data":"15da42bfe77b2ac16e5f976ef96e799820f4ce21980d23af07998d460476e871"}
Sep 09 06:26:34.666169 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:34.665815    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:34.682928 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:34.682883    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d" podStartSLOduration=1.3685620090000001 podStartE2EDuration="1.682868817s" totalImagesPullingTime="314.306808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:26:34.682176257 +0000 UTC m=+2221.898506143" watchObservedRunningTime="2026-09-09 06:26:34.682868817 +0000 UTC m=+2221.899198700"
Sep 09 06:26:40.677956 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:40.677923    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6489fd66cd-7xv5d"
Sep 09 06:26:59.767246 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:59.767150    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5945888b8-vv4kz_95c8fb26-6722-49a2-88a3-39d52db0c0c2/manager/0.log"
Sep 09 06:26:59.990537 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:26:59.990459    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gck77_c6c1cf4c-1991-4869-b663-781a5cc7fbc0/manager/0.log"
Sep 09 06:27:00.221923 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:00.221896    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76b47fccc7-cjspw_b6193bc5-a652-497f-b53f-8fa768a17fcc/copy-manifests/0.log"
Sep 09 06:27:00.227810 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:00.227788    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76b47fccc7-cjspw_b6193bc5-a652-497f-b53f-8fa768a17fcc/manager/0.log"
Sep 09 06:27:00.696603 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:00.696572    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6489fd66cd-7xv5d_9c74b318-d35d-4166-ade9-c56e4079a2e6/maas-api/0.log"
Sep 09 06:27:01.983917 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:01.983885    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-wh9vb_bf928072-5a5e-42e4-9fde-0981e8b8c14d/manager/0.log"
Sep 09 06:27:02.096783 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:02.096754    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-rzldn_bbe0815c-476c-4eba-802a-a67324431747/manager/0.log"
Sep 09 06:27:02.317453 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:02.317356    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-85w5c_d47458d2-76f6-4db9-a2e8-8dec382bbf8e/manager/0.log"
Sep 09 06:27:02.903402 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:02.903375    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs_d3616474-7c24-43cf-bfea-ee196983f08f/istio-proxy/0.log"
Sep 09 06:27:03.014973 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:03.014937    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4mscg_edf95dbf-0662-4437-9a94-95c165b05069/discovery/0.log"
Sep 09 06:27:03.233442 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:03.233350    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f4d448d7f-hj2jf_85a77e83-90c0-41b3-8f32-7e2013c06f2a/kube-auth-proxy/0.log"
Sep 09 06:27:03.462945 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:03.462910    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6fltp_dddcc55b-5388-47fa-bdae-00c80c98d4a4/payload-pre-processing/0.log"
Sep 09 06:27:03.689060 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:03.689030    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78695b457d-lfsxh_08804660-c36f-432a-80c1-6baba82c301f/router/0.log"
Sep 09 06:27:04.028876 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:04.028795    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-hxhv7_b29a44b5-1e98-4569-abf1-aaa167a7ea2e/main/0.log"
Sep 09 06:27:04.132556 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:04.132526    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-bbqs4_e3044f77-f03a-4d81-8881-efce640da8a8/main/0.log"
Sep 09 06:27:04.355000 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:04.354972    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-rrmmf_14b9245f-4729-42eb-8ed4-112a6f5a3490/main/0.log"
Sep 09 06:27:12.666880 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:12.666813    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h8x9t_9e1d1039-4eb0-4691-bd7f-21dfdd3870ca/global-pull-secret-syncer/0.log"
Sep 09 06:27:12.812776 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:12.812737    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tz58z_3c67f7d9-3ba8-43ad-bc02-a063ab1f8c91/konnectivity-agent/0.log"
Sep 09 06:27:12.855940 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:12.855913    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-250.ec2.internal_b918f93a5380fe0ffc07a388ced93b00/haproxy/0.log"
Sep 09 06:27:17.415407 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:17.415374    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-wh9vb_bf928072-5a5e-42e4-9fde-0981e8b8c14d/manager/0.log"
Sep 09 06:27:17.448096 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:17.448059    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-rzldn_bbe0815c-476c-4eba-802a-a67324431747/manager/0.log"
Sep 09 06:27:17.517181 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:17.517159    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-85w5c_d47458d2-76f6-4db9-a2e8-8dec382bbf8e/manager/0.log"
Sep 09 06:27:19.008135 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:19.008095    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15feeca5-4318-462d-a202-84fde3be6099/alertmanager/0.log"
Sep 09 06:27:19.028453 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:19.028430    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15feeca5-4318-462d-a202-84fde3be6099/config-reloader/0.log"
Sep 09 06:27:19.050254 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:19.050233    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15feeca5-4318-462d-a202-84fde3be6099/kube-rbac-proxy-web/0.log"
Sep 09 06:27:19.071476 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:19.071455    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15feeca5-4318-462d-a202-84fde3be6099/kube-rbac-proxy/0.log"
Sep 09 06:27:19.091753 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:19.091729    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15feeca5-4318-462d-a202-84fde3be6099/kube-rbac-proxy-metric/0.log"
Sep 09 06:27:19.113300 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:19.113279    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15feeca5-4318-462d-a202-84fde3be6099/prom-label-proxy/0.log"
Sep 09 06:27:19.133462 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:19.133446    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15feeca5-4318-462d-a202-84fde3be6099/init-config-reloader/0.log"
Sep 09 06:27:19.171019 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:19.170997    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-prgks_f04421dd-ed42-46c3-997e-9bf3d4359d7a/cluster-monitoring-operator/0.log"
Sep 09 06:27:19.329304 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:19.329238    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8kmtm_351a845d-670b-465d-a3ce-2ac23704a55d/node-exporter/0.log"
Sep 09 06:27:19.350333 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:19.350313    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8kmtm_351a845d-670b-465d-a3ce-2ac23704a55d/kube-rbac-proxy/0.log"
Sep 09 06:27:19.369270 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:19.369250    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8kmtm_351a845d-670b-465d-a3ce-2ac23704a55d/init-textfile/0.log"
Sep 09 06:27:19.542264 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:19.542239    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-hmsjp_1b589342-531d-4661-8840-f9a74a79ae59/kube-rbac-proxy-main/0.log"
Sep 09 06:27:19.576742 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:19.576719    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-hmsjp_1b589342-531d-4661-8840-f9a74a79ae59/kube-rbac-proxy-self/0.log"
Sep 09 06:27:19.595709 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:19.595649    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-hmsjp_1b589342-531d-4661-8840-f9a74a79ae59/openshift-state-metrics/0.log"
Sep 09 06:27:21.723370 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:21.723338    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-7lfm4_f163cf03-e312-4296-8fd6-cc95260b51b3/networking-console-plugin/0.log"
Sep 09 06:27:22.316119 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:22.316086    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/1.log"
Sep 09 06:27:22.322286 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:22.322263    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-5mbcw_207d756d-5ccd-456e-bd3f-3ede12b60f78/console-operator/2.log"
Sep 09 06:27:22.829535 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:22.829499    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"]
Sep 09 06:27:22.833445 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:22.833421    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:22.835778 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:22.835755    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kn2d7\"/\"kube-root-ca.crt\""
Sep 09 06:27:22.835897 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:22.835806    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kn2d7\"/\"openshift-service-ca.crt\""
Sep 09 06:27:22.836607 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:22.836587    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kn2d7\"/\"default-dockercfg-z6lpm\""
Sep 09 06:27:22.843212 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:22.843186    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"]
Sep 09 06:27:22.896332 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:22.896306    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54f94d85f7-dqg4q_f86d8f73-4276-413e-9890-686f31835873/console/0.log"
Sep 09 06:27:22.942645 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:22.942616    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rljtw\" (UniqueName: \"kubernetes.io/projected/1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61-kube-api-access-rljtw\") pod \"perf-node-gather-daemonset-5mcp6\" (UID: \"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:22.942816 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:22.942651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61-lib-modules\") pod \"perf-node-gather-daemonset-5mcp6\" (UID: \"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:22.942816 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:22.942721    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61-podres\") pod \"perf-node-gather-daemonset-5mcp6\" (UID: \"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:22.942816 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:22.942792    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61-sys\") pod \"perf-node-gather-daemonset-5mcp6\" (UID: \"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:22.942968 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:22.942834    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61-proc\") pod \"perf-node-gather-daemonset-5mcp6\" (UID: \"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:23.043619 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.043579    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61-sys\") pod \"perf-node-gather-daemonset-5mcp6\" (UID: \"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:23.043803 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.043632    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61-proc\") pod \"perf-node-gather-daemonset-5mcp6\" (UID: \"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:23.043803 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.043693    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rljtw\" (UniqueName: \"kubernetes.io/projected/1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61-kube-api-access-rljtw\") pod \"perf-node-gather-daemonset-5mcp6\" (UID: \"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:23.043803 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.043697    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61-sys\") pod \"perf-node-gather-daemonset-5mcp6\" (UID: \"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:23.043803 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.043712    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61-lib-modules\") pod \"perf-node-gather-daemonset-5mcp6\" (UID: \"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:23.043803 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.043768    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61-podres\") pod \"perf-node-gather-daemonset-5mcp6\" (UID: \"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:23.043803 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.043778    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61-proc\") pod \"perf-node-gather-daemonset-5mcp6\" (UID: \"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:23.044040 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.043879    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61-podres\") pod \"perf-node-gather-daemonset-5mcp6\" (UID: \"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:23.044040 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.043895    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61-lib-modules\") pod \"perf-node-gather-daemonset-5mcp6\" (UID: \"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:23.051184 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.051157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rljtw\" (UniqueName: \"kubernetes.io/projected/1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61-kube-api-access-rljtw\") pod \"perf-node-gather-daemonset-5mcp6\" (UID: \"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:23.145039 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.144956    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:23.415102 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.415077    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"]
Sep 09 06:27:23.416359 ip-10-0-131-250 kubenswrapper[2564]: W0909 06:27:23.416328    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1bfdc459_6d5e_40d7_bd7a_bc16f9c26a61.slice/crio-75a9c4807ccdeb17a1874df6b62a98164d0c9e26562878051ac85d208226d210 WatchSource:0}: Error finding container 75a9c4807ccdeb17a1874df6b62a98164d0c9e26562878051ac85d208226d210: Status 404 returned error can't find the container with id 75a9c4807ccdeb17a1874df6b62a98164d0c9e26562878051ac85d208226d210
Sep 09 06:27:23.616592 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.616561    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-qb2nd_be5d31e4-20fc-48c9-935d-087f65b77eca/volume-data-source-validator/0.log"
Sep 09 06:27:23.956570 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.956526    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6" event={"ID":"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61","Type":"ContainerStarted","Data":"d262e295e62758a7a7bd34d4dedbd74150903c91f9682f094659bebb9b89524a"}
Sep 09 06:27:23.956931 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.956590    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6" event={"ID":"1bfdc459-6d5e-40d7-bd7a-bc16f9c26a61","Type":"ContainerStarted","Data":"75a9c4807ccdeb17a1874df6b62a98164d0c9e26562878051ac85d208226d210"}
Sep 09 06:27:23.956931 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.956613    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:23.972203 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:23.972159    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6" podStartSLOduration=1.972144009 podStartE2EDuration="1.972144009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:27:23.97098704 +0000 UTC m=+2271.187316925" watchObservedRunningTime="2026-09-09 06:27:23.972144009 +0000 UTC m=+2271.188473892"
Sep 09 06:27:24.754229 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:24.754197    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pvjjm_9f3f129d-c477-4771-a0cd-4c9e710bb832/dns/0.log"
Sep 09 06:27:24.775241 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:24.775220    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pvjjm_9f3f129d-c477-4771-a0cd-4c9e710bb832/kube-rbac-proxy/0.log"
Sep 09 06:27:24.860178 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:24.860146    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h8v7x_40030ec2-6b02-4d13-b794-aa551a09290e/dns-node-resolver/0.log"
Sep 09 06:27:25.538177 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:25.538148    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dmx8r_f4234e30-8f0b-47d1-a67d-36deddc3a89c/node-ca/0.log"
Sep 09 06:27:26.443061 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:26.443029    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs_d3616474-7c24-43cf-bfea-ee196983f08f/istio-proxy/0.log"
Sep 09 06:27:26.699747 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:26.699666    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4mscg_edf95dbf-0662-4437-9a94-95c165b05069/discovery/0.log"
Sep 09 06:27:26.749995 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:26.749970    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f4d448d7f-hj2jf_85a77e83-90c0-41b3-8f32-7e2013c06f2a/kube-auth-proxy/0.log"
Sep 09 06:27:26.827700 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:26.827669    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6fltp_dddcc55b-5388-47fa-bdae-00c80c98d4a4/payload-pre-processing/0.log"
Sep 09 06:27:26.881432 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:26.881403    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78695b457d-lfsxh_08804660-c36f-432a-80c1-6baba82c301f/router/0.log"
Sep 09 06:27:27.716812 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:27.716768    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9gc29_be276cf4-4edd-4f8e-a2fa-1c2e9deed8f8/serve-healthcheck-canary/0.log"
Sep 09 06:27:28.188386 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:28.188354    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-cflnz_d1b76aac-f159-4940-8a85-64a227c18b05/insights-operator/0.log"
Sep 09 06:27:28.188579 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:28.188418    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-cflnz_d1b76aac-f159-4940-8a85-64a227c18b05/insights-operator/1.log"
Sep 09 06:27:28.274326 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:28.274300    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hc2v2_23b784c5-0b83-493f-917e-559bc6a26b72/kube-rbac-proxy/0.log"
Sep 09 06:27:28.293240 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:28.293215    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hc2v2_23b784c5-0b83-493f-917e-559bc6a26b72/exporter/0.log"
Sep 09 06:27:28.313508 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:28.313478    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hc2v2_23b784c5-0b83-493f-917e-559bc6a26b72/extractor/0.log"
Sep 09 06:27:29.972508 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:29.972465    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-5mcp6"
Sep 09 06:27:30.676529 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:30.676471    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5945888b8-vv4kz_95c8fb26-6722-49a2-88a3-39d52db0c0c2/manager/0.log"
Sep 09 06:27:30.775318 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:30.775285    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gck77_c6c1cf4c-1991-4869-b663-781a5cc7fbc0/manager/0.log"
Sep 09 06:27:30.846933 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:30.846910    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76b47fccc7-cjspw_b6193bc5-a652-497f-b53f-8fa768a17fcc/manager/0.log"
Sep 09 06:27:30.865784 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:30.865755    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76b47fccc7-cjspw_b6193bc5-a652-497f-b53f-8fa768a17fcc/copy-manifests/0.log"
Sep 09 06:27:32.434218 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:32.434187    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-vk5zx_0b37f545-4ed7-44e5-be4d-9de2f35425bd/openshift-lws-operator/0.log"
Sep 09 06:27:32.951302 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:32.951214    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs_d3616474-7c24-43cf-bfea-ee196983f08f/istio-proxy/0.log"
Sep 09 06:27:33.199558 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:33.199529    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4mscg_edf95dbf-0662-4437-9a94-95c165b05069/discovery/0.log"
Sep 09 06:27:33.229247 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:33.229177    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f4d448d7f-hj2jf_85a77e83-90c0-41b3-8f32-7e2013c06f2a/kube-auth-proxy/0.log"
Sep 09 06:27:33.277622 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:33.277592    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6fltp_dddcc55b-5388-47fa-bdae-00c80c98d4a4/payload-pre-processing/0.log"
Sep 09 06:27:33.317574 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:33.317547    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78695b457d-lfsxh_08804660-c36f-432a-80c1-6baba82c301f/router/0.log"
Sep 09 06:27:34.809337 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:34.809309    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xdtqs_d3616474-7c24-43cf-bfea-ee196983f08f/istio-proxy/0.log"
Sep 09 06:27:35.064250 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:35.064176    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4mscg_edf95dbf-0662-4437-9a94-95c165b05069/discovery/0.log"
Sep 09 06:27:35.102711 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:35.102686    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f4d448d7f-hj2jf_85a77e83-90c0-41b3-8f32-7e2013c06f2a/kube-auth-proxy/0.log"
Sep 09 06:27:35.155555 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:35.155530    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6fltp_dddcc55b-5388-47fa-bdae-00c80c98d4a4/payload-pre-processing/0.log"
Sep 09 06:27:35.196486 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:35.196455    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78695b457d-lfsxh_08804660-c36f-432a-80c1-6baba82c301f/router/0.log"
Sep 09 06:27:37.852325 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:37.852297    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-rghsp_c2ce177a-74d8-4240-8eb1-bf8c703e59d6/kube-storage-version-migrator-operator/1.log"
Sep 09 06:27:37.853189 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:37.853170    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-rghsp_c2ce177a-74d8-4240-8eb1-bf8c703e59d6/kube-storage-version-migrator-operator/0.log"
Sep 09 06:27:39.264953 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:39.264924    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wrz2h_9d280b22-a809-4323-8a18-2049d0394312/kube-multus-additional-cni-plugins/0.log"
Sep 09 06:27:39.285835 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:39.285812    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wrz2h_9d280b22-a809-4323-8a18-2049d0394312/egress-router-binary-copy/0.log"
Sep 09 06:27:39.306346 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:39.306324    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wrz2h_9d280b22-a809-4323-8a18-2049d0394312/cni-plugins/0.log"
Sep 09 06:27:39.326299 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:39.326280    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wrz2h_9d280b22-a809-4323-8a18-2049d0394312/bond-cni-plugin/0.log"
Sep 09 06:27:39.345248 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:39.345233    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wrz2h_9d280b22-a809-4323-8a18-2049d0394312/routeoverride-cni/0.log"
Sep 09 06:27:39.364630 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:39.364613    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wrz2h_9d280b22-a809-4323-8a18-2049d0394312/whereabouts-cni-bincopy/0.log"
Sep 09 06:27:39.384854 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:39.384836    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wrz2h_9d280b22-a809-4323-8a18-2049d0394312/whereabouts-cni/0.log"
Sep 09 06:27:39.460684 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:39.460657    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xlfg6_0b4fc8ea-c8ee-42aa-b597-b3eb70edb43c/kube-multus/0.log"
Sep 09 06:27:39.481405 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:39.481380    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qgb26_910ab0f9-8007-4ec5-b5b9-bf0e32c85593/network-metrics-daemon/0.log"
Sep 09 06:27:39.499004 ip-10-0-131-250 kubenswrapper[2564]: I0909 06:27:39.498981    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qgb26_910ab0f9-8007-4ec5-b5b9-bf0e32c85593/kube-rbac-proxy/0.log"